Query regarding "partition.getPartitionedInstance()"

[Devavrath S]

Hi All,

I had a quick question regarding the "partition.getPartitionedInstance()" function.

Scenario: I have to set certain properties of an already existing, running operator when the definePartitions() function is called. 

I am using the above mentioned function ("partition.getPartitionedInstance()") to set the properties but the operator is not picking up these properties in the next window ("begin window"). 

Am I doing this right? 

Is there another workaround? 

Thank you in advance!!

Regards,

Devavrath S

[Sanjay Pujare]

Hi Devavrath

Looking at the Partitioner code I see this comment:

The list of existing partitions reflects the last checkpoint state for each of the operator instances. When

   * creating new partitions, the implementation has the opportunity to transfer state from these existing operator

   * instances to new instances. At minimum, properties set at initialization time on the original operator need to be

   * set on new instances.

So in your definePartitions even if you don't add/delete any partition you need to return the modified partition in the Collection<Partition<T>> as per the contract. Also how are you modifying the properties of the partition?

Sanjay

--
You received this message because you are subscribed to the Google Groups "DataTorrent Users Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dt-users+unsubscribe@googlegroups.com.
To post to this group, send email to dt-u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dt-users/CALzAcj9gNCywEiB9MuwdcZWtaThoJA1XmTLVJ%2B2SQGVPzUArFw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Vlad Rozov]

Hi Devavrath,

Can you provide more details on your use case? What operator properties do you need to change?

Thank you,

Vlad

[Devavrath S]

Hi Vlad and Sanjay,

I thank you for your response.

I would like to reiterate myself. I have a couple of variables (flags) for the operator with getters and setters. 

When the definepartitions() is called the first time for the operator, I have the flags set, by default as false.

The definepartitions() is triggered again from processStats(), after the occurrence of a particular condition. At this point in time, I would like to set the the flags to "true" and add it to the Collection<Partition<T>>. The operator in it's lifecycle has to pick up these flags and act accordingly. 

Flow is as follows: 

• Operator Code :  

public class DownStreamOperator implements InputOperator, Operator.ActivationListener<Context> {

    private boolean shouldWeStartConnToUpstreamOperator;

    

    public DownStreamOperator() {

        this. shouldWeStartConnToUpstreamOperator = false;

    }

    public boolean getShouldWeStartConnToUpstreamOperator() {

        return shouldWeStartConnToUpstreamOperator;

    }

    public void setshouldWeStartConnToUpstreamOperator(boolean shouldWeStartConnToUpstreamOperator) {

        this.shouldWeStartConnToUpstreamOperator = shouldWeStartConnToUpstreamOperator;

    }

}

• Partitioner : 

    for (Partition<DownStreamOperator> partition : partitions) {

DownStreamOperator oper = partition.getPartitionedInstance();

partition.getPartitionedInstance().setshouldWeStartConnToUpstreamOperator(true);

newPartitions.add(partition);

    }

   return newPartitions;

I am not sure if I am doing something wrong. Kindly let me know if this is right.

Thanks in advance!!

Regards,

Devavrath S

[Vlad Rozov]

Hi Devavrath,

The way you set the operator property only affects the operator instance that runs inside the Application Master. To set the property inside containers you will need to send a request to the Application Master to change the property and I am not sure that this will be the best solution for your use case. For us to be able to help, please provide more details how you plan to use the flag. Can you also share the problem or use case for your Partitioner.

Thank you,

Vlad