How does Narayana Recovery work in OpenShift?

[Gabsaga Tata]

I have version 3+ of Spring Boot for my application and I am using Narayana snowdrop for XA transactions. In my Java config, I have the recovery manager set up as follows which I can see it running periodically using default configuration.

@Bean(initMethod = "start", destroyMethod = "stop")
public RecoveryManagerService recoveryManagerService() {
RecoveryManagerService recoveryManagerService = new RecoveryManagerService();
recoveryManagerService.create();
return recoveryManagerService;
}

When I deploy to OpenShift, my application has 3 replicas (pods) running. How can I make sure that one of the pod's recovery manager is running at any given time to avoid collision in recovery?

Thanks,
Gabsaga

[Manuel Finelli]

Hi Gabsaga,

Thanks for your email.

Unfortunately, we don't currently support Snowdrop in OpenShift out of the box. Users need to ensure that Narayana's requirements are met, such as maintaining a globally unique Transaction Manager ID and only having one Recovery Manager per Object Store. Having said that, I worked on ArjunaCore's graceful shutdown to keep the Recovery Manager running until all transactions are recovered (JBTM-3894). However, we're not yet at a point where Narayana can offer a complete solution that works across all cloud environments and lifts the burden from users to meet Narayana's requirements. Of course, I'm happy to discuss/support our community if you or others would like to contribute to this work and get Snowdrop running in OpenShift or another cloud environment.

FYI, we developed a module in the WildFly Kubernetes Operator to simulate a graceful shutdown for WFLY pods. To solve some of the challenges Narayana presents in cloud environments, we used OpenShift StatefulSets in WFLY K8s Operator. Since then, part of the logic we developed in the WFLY operator went into JBTM-3894.

Best wishes,

Manuel

--
You received this message because you are subscribed to the Google Groups "narayana-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to narayana-user...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/narayana-users/120e2af4-33cc-4efa-addd-ca8a9423c943n%40googlegroups.com.

[Gabsaga Tata]

Thank you Manuel for the prompt response. Here is what I am planning on doing.

I have the following snippet as shown below and I am using StatefulSet PVC with ReadWriteOnce access mode.
My understand is that with a StatefulSet, the pod names will have an ordinal starting at index 0.
For example: pod-abcde-0, pod-abcde-1, pod-abcde-2 and with my setup the log directory for the pods will be:
/base/log/dir/pod-abcde-0/base/log/dir/pod-abcde-0
/base/log/dir/pod-abcde-1/base/log/dir/pod-abcde-0
/base/log/dir/pod-abcde-2/ObjectStore.....

And so if say pod-abcde-1 crashes for some reason, the controller will restart it with same name and ordinal since
it's a StatefulSet which will be pod-abcde-1 and therefore will be able to read and recover from same log directory.

I am hoping this will work.

@Configuration
public class NarayanaCorePropertiesConfig {

  @PostConstruct
  public void setNodeIdentifier() throws CoreEnvironmentBeanException {
    com.arjuna
        .ats
        .arjuna
        .common
        .arjPropertyManager
        .getCoreEnvironmentBean()
        .setNodeIdentifier(NarayanaUtil.getNodeIdentifier());
  }
}

  @PostConstruct
  public void setTransactionLogDir() {
    String podLogDir = baseTransLogDir + "/" + NarayanaUtil.getNodeIdentifier();
    System.setProperty("user.dir", podLogDir);
    System.setProperty("ObjectStoreEnvironmentBean.objectStoreDir", podLogDir);
  }

public class NarayanaUtil {

  private NarayanaUtil() {}

  private static final String DEFAULT_HOST = "localhost-v1";

  public static String getNodeIdentifier() {
    String podName = System.getenv().getOrDefault("HOSTNAME", DEFAULT_HOST);
    if (DEFAULT_HOST.equals(podName)) {
      return podName;
    }

    // Remove last 17 characters from the pod name
    String newPodName = podName.substring(0, podName.length() - 17);

    // Check if the new pod name's length is greater than 28
    if (newPodName.length() > 28) {
      // If greater, return the first 28 characters
      return newPodName.substring(0, 28);
    }
    return newPodName;
  }
}

[Manuel Finelli]

Hi Gabsaga,

Thanks for reaching out. Your approach seems very promising.

Although I don't have much experience with Spring Boot or Snowdrop, I have a couple of suggestions from a Narayana point of view that might be helpful:

* It's important to make sure that `NarayanaUtil.getNodeIdentifier()` provides a unique identifier for each pod instance. If multiple Narayana instances have the same ID, you could have issues with transaction recovery

* You should also plan how your Narayana instances handle scale-down events. If a pod is terminated while transactions are still in flight or in need of recovery, they can be left in an inconsistent state. Consider integrating the new graceful shutdown feature, developed in JBTM-3894, into your pod's pre-stop lifecycle hook. This should allow Narayana's recovery manager to complete its work before the pod is terminated

Best wishes,

Manuel

To view this discussion, visit https://groups.google.com/d/msgid/narayana-users/d4648ba7-364c-4413-9cf4-db9837e6319en%40googlegroups.com.

[Manuel Finelli]

Hi Gabsaga,

Do you have any further questions or require any more input from me? Otherwise, we can go ahead and mark this thread as completed.

Thanks.

BW,

Manuel