Hourglass Input paths

[Abhishek Gayakwad]

Hi,

Hourglass requires input directory structure in yyyy/mm/dd format, Is it possible to provide custom logic for reading from different directory structure e.g. yyyymmdd or some other date key. Is it possible to change the frequency form daily to weekly ?

Thanks

Abhishek

[Matthew Hayes]

Hi Abhishek,

At the moment the input directory structure is fixed to the yyyy/mm/dd format.  This is one area where I'd like to make the system more customizable.  I'd like to create an interface that represents the concept of a set of input paths that each are associated with a particular date range, but without being tied to a particular format like yyyy/mm/dd.  Then you would have the option of creating a class that implements this interface and providing this class to Hourglass.  You could create a class like WeeklyInput for example that assumes yyyy/mm/dd structure but dates only on week boundaries.  Or you could create a class like MonthlyInput that uses yyyy/mm.  Or one that uses yyyymmdd.  The same could be done for the output path.  You could even have an input structure that has overlapping date ranges, like "daily/yyyy/mm/dd" and "monthly/yyyy/mm".  Hourglass could determine the appropriate files to read based on the desired date range to consume.  These are just some ideas.  I haven't had time to work on this yet.  If you're interested in working on it we would appreciate the help ;)

-Matt

--
You received this message because you are subscribed to the Google Groups "DataFu" group.
To unsubscribe from this group and stop receiving emails from it, send an email to datafu+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.