Loading the properties file at run time in DT Application.

[PULLARAO KOTA]

Hi, 
I have a DataTorrent application which loads the properties from file and use them in code, and some times we have to change the properties at regular intervals, in this case every time we have to stop and start the Application. This will be a hectic job restart the Application each time. Is there any method how to read properties at run time even after we change the file?

[Yogi Devendra]

Hi,

I assume you are referring to operator properties loaded from xml or json configuration.

1. dtcli provides a mechanism to connect to the running application using following command:

connect app-id

Connect to an app

2. Following command is available after you connect the application to set the operator properties.

set-operator-property operator-name property-name property-value

Set a property of an operator

~ Yogi

P. S: malhar...@googlegroups.com is now deprecated. 

 Kindly use us...@apex.incubator.apache.org mailing list for any further  questions. 

On Wed, Dec 30, 2015 at 3:04 PM, PULLARAO KOTA <10fe1...@gmail.com> wrote:

Hi, 
I have a DataTorrent application which loads the properties from file and use them in code, and some times we have to change the properties at regular intervals, in this case every time we have to stop and start the Application. This will be a hectic job restart the Application each time. Is there any method how to read properties at run time even after we change the file?

--
You received this message because you are subscribed to the Google Groups "Malhar" group.
To unsubscribe from this group and stop receiving emails from it, send an email to malhar-users...@googlegroups.com.
To post to this group, send email to malhar...@googlegroups.com.
Visit this group at https://groups.google.com/group/malhar-users.
For more options, visit https://groups.google.com/d/optout.

[PULLARAO KOTA]

Hi I am not using Dtcli,Using the DataTorrent Console Web UI To Upload Jar and Run the Application i want to load the properties at the time of Launching the application instead of giving each property separately.

The Properties file will Be As Key=Value pairs.Please Suggest a method to solve the problem.

[Yogi Devendra]

1. Web UI has facility to manually change the operator properties.

2. If you need to change it programmatically you can have a look at REST API documentation available here: http://docs.datatorrent.com/dtgateway_api/ 

This is like 'push'  the properties whenever there is some change.

3. If you are looking for continuous polling of some properties file where some external program is writing property values:

You can make use of handleIdleTime() from in your operator code and achieve this.

But, I would recommend option 2 over option 3.

~ Yogi

[Bhupesh Chawda]

Hi,

In case of the DataTorrent console web UI, you can add an xml config file to the resources directory of your application project.
This will allow you to use the "use config file" option on the web UI when launching the application.

Thanks.

Bhupesh

[Sandeep Deshmukh]

For 3, handleIdleTime() is not the right thing for this requirement. This function call is not guaranteed. You can use beginWindow or endWindow for such purpose.

Regards,

Sandeep

On Wed, Dec 30, 2015 at 6:02 PM, Yogi Devendra <deve...@datatorrent.com> wrote:

[t...@datatorrent.com]

Yogi, I would not recommend pushing properties using the rest api. Currently properties are applied in the middle of an application window and it is not fault tolerant. I would also not recommend using handleIdleTime as Sandeep as mentioned. 

Pullarao, the safest mechanism to apply multiple properties at the same time, while the application is running, is to do the following:

- Add a property update operator to your dag which polls the properties.

- Connect the property update operator to all the operators in your dag with input port / output port / stream which need to be updated.

- When updates are received for properties, send the relevant updates to each operator.

- When an operator receives an update, hold it until the beginning of the next window and apply the property in beginWindow.

- Make sure all the operators have the same application window count.

This is a very roundabout way to do it, and requires the application developer to write code, but it is the only fault tolerant way to do things as of now. There is a ticket open to set multiple properties at once on operators safely in one REST call or cli call. That feature is currently on the apex roadmap.

https://issues.apache.org/jira/browse/APEXCORE-163

Thanks,

Tim