TimeSensor with timezone

[John Garnica]

Hi,

I would like to use TimeSensor with a specified timezone. I looks like TimeSensor is ok to receive a python datetime.timedelta object which does have an attribute of tzinfo (see https://docs.python.org/2/library/datetime.html).

However when passing in the timezone, it still triggered on the Server Time (in my instance Sydney Australia time) instead of the intended Timezone.

That is 5pm Sydney time, not 5pm New York time. See example below

WaitForNewYorkEODStartTime = TimeSensor(
    task_id='WaitForNewYorkEODStartTime',
    target_time=time(hour=17, minute=0, second=0, microsecond=0, tzinfo=timezone('America/New_York')),
    dag=dag
)

Should this be supported?

Log below:

[2015-10-29 00:00:14,102] {models.py:107} INFO - Filling up the DagBag from /home/calypso/calypso/airflow/dags/simpleeod.py
[2015-10-29 00:00:14,102] {models.py:174} INFO - Importing /home/calypso/calypso/airflow/dags/simpleeod.py
[2015-10-29 00:00:14,261] {models.py:226} INFO - Loaded DAG <DAG: simpleeod>
[2015-10-29 00:00:17,574] {models.py:107} INFO - Filling up the DagBag from /home/calypso/calypso/airflow/dags/simpleeod.py
[2015-10-29 00:00:17,574] {models.py:174} INFO - Importing /home/calypso/calypso/airflow/dags/simpleeod.py
[2015-10-29 00:00:17,736] {models.py:226} INFO - Loaded DAG <DAG: simpleeod>
[2015-10-29 00:00:17,777] {models.py:874} INFO - 
--------------------------------------------------------------------------------
New run starting @2015-10-29T00:00:17.774897
--------------------------------------------------------------------------------
[2015-10-29 00:00:17,785] {models.py:907} INFO - Executing <Task(TimeSensor): WaitForNewYorkEODStartTime> on 2015-10-28 00:00:00
[2015-10-29 00:00:17,785] {sensors.py:357} INFO - Checking if the time (17:00:00) has come
[2015-10-29 00:01:17,784] {sensors.py:357} INFO - Checking if the time (17:00:00) has come

...

[2015-10-29 16:59:17,983] {sensors.py:357} INFO - Checking if the time (17:00:00) has come
[2015-10-29 17:00:17,983] {sensors.py:357} INFO - Checking if the time (17:00:00) has come
[2015-10-29 17:00:17,984] {sensors.py:57} INFO - Success criteria met. Exiting.

Thanks in Advance.
John

[Garry van Diggele]

We are interested in this functionality as well. Reading the code and documentation in detail. 

class TimeSensor(BaseSensorOperator):
    """
    Waits until the specified time of the day.

    :param target_time: time after which the job succeeds
    :type target_time: datetime.time
    """
    template_fields = tuple()

    @apply_defaults
    def __init__(self, target_time, *args, **kwargs):
        super(TimeSensor, self).__init__(*args, **kwargs)
        self.target_time = target_time

    def poke(self, context):
        logging.info(
            'Checking if the time ({0}) has come'.format(self.target_time))
        return datetime.now().time() > self.target_time

Fundamentally the mehod datetime.now() Without a TZ object ignores the Time Zone of the machine. To actually implement a real timesensor the correct implementation would be:

datetime.now().time(get_local_zone) 

Is this correct? The only issue I really see is then the date ticks over to the next day we actually need to compare a DAteTime object. Compare it to the Execution Date/Time for the sensor. As such the implementation would need to be changed quite a bit. Fundamentally a TimeSensor should be TZ aware.

Any one have any thoughts on this? 

[Maxime Beauchemin]

The way we think about timezones at Airbnb is everything is in UTC everywhere (servers and timestamps in logs/databases). Though some UIs will localize. Pretty much all engineers here build an internal clock for UTC, knowing at all time what UTC time it is.

Of course I'm happy reviewing PR adding functionality around timezones in Airflow, as long as the API doesn't change for people who don't want to use those features.

Max

[Garry van Diggele]

So how does AirBnB handle jobs which are daylight time sensitive if your scheduling jobs for UTC times?