Static PEPs, Delay Rules, and iRODS v4.2.9

[Kory Draughn]

Hello Everyone,

Below the dotted line you'll find the contents of a new iRODS RFC.  We think it is important for the community to have

a chance to voice their opinions and concerns about the changes mentioned in the RFC.  We appreciate you taking the time
to read this and we hope to hear from you soon.

You can find the RFC at https://github.com/irods/irods_rfcs/blob/master/0006_static_peps_and_delay_rules.md

Thanks,

Kory Draughn
iRODS Consortium
-----------------------------------------------------------------------------------------------------------------

- Feature Name: Static PEPs and Delay Rules
- Status: in-progress
- Start Date: 2020-10-13
- Authors: Kory Draughn
- RFC PR: [#5175](https://github.com/irods/irods/pull/5175)
- iRODS Issue: [#3049](https://github.com/irods/irods/issues/3049), [#4428](https://github.com/irods/irods/issues/4428), [#5153](https://github.com/irods/irods/issues/5153)

# Static PEPs and Delay Rules

## What is this about?
iRODS v4.2.9 contains a lot of changes. Some of the most important and exciting changes have to do with
Intermediate Replicas, Logical Locking, and the Rule Execution (RE) Delay Server. In particular, this thread is
interested in the community's thoughts about the changes to the RE Delay Server.

## What changed?
In previous versions of iRODS, delay rules involved storing the contextual information in a file on the local
hard drive. These files are known as Rule Execution Information files or REI files for short. These files contained
information about who scheduled the rule, auth info, proxy info, data object info, and all sorts of information needed
to successfully execute the rule later. This mostly worked, but presented problems related to performance and hard
drive space. There are also issues with migrating the RE delay server to a different machine.

To get around these issues, iRODS v4.2.9 has moved storage of the contextual information from a file on the local
hard drive to the catalog. The contextual information is stored in a new column (i.e. `r_rule_exec.exe_context`)
as JSON. This solves a multitude of problems. This allows the RE delay server to be migrated without any issues
while also (hopefully) improving performance.

## Are there any downsides to this change?
This should not be viewed as a downside. These changes are a step to future proofing your deployment.

Aside from that, the most important thing to know in regards to these changes is that Static PEPs can no longer be used
to schedule delay-based rules. For users and administrators who are using Static PEPs to schedule delay-based rules,
you will need to update your policy to use Dynamic PEPs instead.

We chose to not add support for Static PEPs due to the following reasons:
- It helps push the community towards Dynamic PEPs.
- It enables a path to deprecating and removing Static PEPs.
- Static PEPs are planned for removal in iRODS v4.3.0.
- Attempting to support Static PEPs meant we'd have to serialize ALL session variables.

Given those reasons, it did not make sense to add support for them. Especially when the plan is to focus on getting
iRODS v4.3.0 released before UGM 2021.

## Will I have to do anything special to upgrade my deployment?
Hopefully not, iRODS v4.2.9 comes with a migration application that will automatically convert and import all existing REI
files from the local hard drive into the catalog during server startup. The tool will provide information about the
number of files processed as well as a log file for what happened during processing. The tool is also safe to re-run
as many times as you like.

However, the tool was implemented with the assumption that the contextual information was produced by a dynamic PEP.
Another way to handle an upgrade is to drain all delay rules from the catalog. This will effectively skip the need of
the migration tool and allow a clean upgrade.

## The Community
With all of that said, we are interested in hearing what the community thinks of these changes.
- Q. Do you agree with our decisions?
- Q. How will this affect your deployment?
- Q. Is there a better way to approach this?
- Q. Have we covered all scenarios?

## Related Information
See the following for details on what lead to this solution.
- https://github.com/irods/irods/issues/3049
- https://github.com/irods/irods/issues/4428
- https://github.com/irods/irods/issues/5153
- https://github.com/irods/irods/pull/5175

[John Constable]

Hi Kory, all!

Wonderfully detailed and explained as always, thank you. I hope its OK to reply here and not as an issue on the RFC repo?

My Initial thought was about the upgrade tool, where you said;

“iRODS v4.2.9 comes with a migration application that will automatically convert and import all existing REI files from the local hard drive into the catalog during server startup.”

Would it be possible to run this import tool before the upgrade? I’m thinking particularly of reducing any periods where the server/zone is in an in-between state between, say 4.2.8 and 4.2.9, where the upgrade script needs to complete before the service/zone is functional again, particularly if you have a large rule backlog (such as, say, importing 220 million metadata objects into elastic search via the indexing plugin :-) ). The database change appears to add columns rather than amend, if so, this could be done before the upgrade? Once the backlog was converted, any new/in progress rules would be easily then migrated as part of the upgrade in a matter of moments.

If not, can the tool be de-coupled from the upgrade so it could be run at any time after (such as with the 41.x -> 4.2 stale information removal script)? What happens in this case, will the rules continue to be processed, and new rules go into the catalog? If so this seems a safe and low impact path to migrating, especially if it can be stopped/started as required?

It would also be helpful to have a dry run option that would suggest how much additional space would be required in the database, and if any optimisations could be applied, such as indexes, trigrams and so forth. It would be embarrassing to run the database server out of temp table space mid-commit, or other capacity during the upgrade, if dealing with a large rule backlog to migrate!

You also mentioned; “Another way to handle an upgrade is to drain all delay rules from the catalog. This will effectively skip the need of the migration tool and allow a clean upgrade.”

As I understand this, it means let all the outstanding rules run before starting the upgrade and prevent any new one from starting? This would mean an extended downtime for the upgrade, I would presume, as functionality would have to be disabled to allow the backlog to drain?

Lastly, this statement;

“Static PEPs are planned for removal in iRODS v4.3.0.” Is there a list of Static PEPS and their targets that people can start to plan their migrations away, so that its a non-event by 4.3?

Hopefully I’ve not misunderstood something critical in the above, and its of some help.

John

[Kory Draughn]

Hi John,

Based on your feedback, we could avoid downtime and simplify the upgrade process by making the following adjustments:
- The upgrade process would only be responsible for adding a new column to the R_RULE_EXEC table.
- The upgrade process would not attempt to migrate/import any REI files into the catalog.
- The delay server would import REI files into the catalog while processing the associated delay rule.
- Delay rules that use session variables would not be imported and continue to require the existence of an REI file.
- The migration/import tool would be unnecessary and not exist.
- An application would be provided separately so that administrators can detect additional space requirements and other things before actually performing any upgrades.

As for providing a mapping between static and dynamic PEPs, this may be difficult due to the fact that there is no exact mapping between them.  The goals between each entity's deployment can vary and may require different PEPs to achieve the desired result.  Please see https://docs.irods.org/4.2.8/getting_started/upgrading/#migrating-from-static-peps-to-dynamic-peps for more information.

Thoughts?

Kory Draughn
iRODS Consortium

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/c1a6aff6-4bcd-4e4a-b3a8-ce26bc96505an%40googlegroups.com.

[jc...@sanger.ac.uk]

This sounds good to me, thank you!

[Tony Edgin]

Hi Kory.

I'm sorry to take so long to respond. No good excuse, really.

To answer your questions, Thank you! I've been hoping delay rule thunks would be moved into the ICAT DB.

Does static PEPs no longer being supported mean that the delay microservice will fail if called through a static PEP even if called indirectly by a nested rule and even if the session variables are not accessed by the delayed rule? Here's an example.

do_something(*Var) { msiSomething(*Var) }

schedule_something(*Var) {

  delay('<PLUSET>1s</PLUSET><EF>1s</EF>') { do_something(*Var) }

}

acPostProcForStaticPEP { schedule_something($sessionVar) }

If this would no longer work, I'll need to switch a half dozen static PEPs to dynamic ones and modify 40 or so rules. I'm happy to do this. In 4.1, there are some things you can do with static PEPs that can't be done with dynamic ones. I'm hoping this has changed in 4.2. We are beginning the process of upgrading to iRODS 4.2 next month. Depending on when 4.2.9 will be released, we would either upgrade to 4.2.8 or 4.2.9. Knowing that our rule logic will need to be significantly changed for 4.2.9, we would upgrade to 4.2.8 regardless of 4.2.9's release date.

Is it much work to allow a delay rule to be scheduled from a static PEP and just not serialize the session variables? The delay rule would fail if it attempts to use a session variable much like a remote rule currently does.

I hope I don't sound negative. I'm looking forward to this change. Thanks again!

Cheers,

Tony Edgin

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/c5b39169-471f-48b4-932b-3efc40dcfc45n%40googlegroups.com.

[Kory Draughn]

Hi Tony,

Interesting. I hadn't considered that scenario.

FYI, the RFC was recently updated and my response is based on those changes. You can find the RFC here: https://github.com/irods/irods_rfcs/blob/master/0006_static_peps_and_delay_rules.md

Does static PEPs no longer being supported mean that the delay microservice will fail if called through a static PEP even if called indirectly by a nested rule and even if the session variables are not accessed by the delayed rule?

No. This means that the delay microservice will fail if the delay rule's rule text contains at least one session variable. I haven't verified what would happen given an example like the one you presented, but based on the latest RFC implementation, I believe your example would result in the delay rule being stored in the catalog. This is because the actual rule text of the delay rule does not contain any session variables. As for the rule actually running later, that depends on how *Var is captured.

... there are some things you can do with static PEPs that can't be done with dynamic ones ...

Are those things documented in an issue by chance?

Is it much work to allow a delay rule to be scheduled from a static PEP and just not serialize the session variables?

This behavior is already supported. Static PEPs can still schedule delay rules as long as the rule text does not directly contain any session variables. This requirement exists because session variables are not supported by the RFC.

Hopefully that answers your questions.

Please continue to bring up any concerns with the direction of the RFC. The more people who see it, the better :)

Thanks,

Kory

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CACkmaQVhnvFH8iqTWyz3g%2BnYi_TYQ1F7B%3DWjr9Q0DxL%3D8sBSgw%40mail.gmail.com.

[Kory Draughn]

Tony,

After some investigation, I have confirmed that your example works with the current RFC implementation. As long as the delay invocation happens indirectly, everything should be fine.

One thing to note with the current implementation is that session variables aren't allowed anywhere inside of the delay rule, even comments. I'll see if I can lift that restriction before the release.

Kory

[Tony Edgin]

Hi Kory.

This is great news. Thanks.

Since I' running iRODS 4.1.10, I assumed that dynamic PEP coverage was incomplete and didn't report any bugs. After upgrading to 4.2, I'll begin migrating our static PEPs to dynamic ones. I'll let you all know if I think I've found a hole in the coverage.

Cheers,

Tony

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CAA-7h7k%3DJCv7xWs4NyE7BVT1MoRG70owevAHvDALSHjD%3Db-UEA%40mail.gmail.com.