hfs-textline to take multiple input files

[thejus]

Hi Guys,

Here is my problem,

I have a directory which contains dated files.

Every day I need to run a cascalog job on a subset of these files.

I can copy this subset into a temporary directory, run the job and delete it later.

But these files are huge and hence take a considerable amount of time to copy.

As far as i have checked hfs-texline takes only one path.

Is there hfs-textline like function to create a tap for multiple files,

Or a function that can append two taps.

--

Thejus

[Sam Ritchie]

Hey,

you can use :source-pattern like so:

(hfs-textline "/root/path" :source-pattern "/2011/*/01/*")

 to get all data in the first of every month, for example.

-- 

Sam Ritchie, Twitter Inc

703.662.1337

@sritchie

(Too brief? Here's why! http://emailcharter.org)

[thejus]

Hi Sam,

I was looking for the same.

Thanks.

--

Thejus

[Stefan Hübner]

Sam Ritchie <sritc...@gmail.com> writes:

> Hey,
>
> you can use :source-pattern like so:
>
> (hfs-textline "/root/path" :source-pattern "/2011/*/01/*")
>
> to get all data in the first of every month, for example.

Actually, adding the glob pattern to the path works just as well.

[Erik Andrejko]

Hi Sam,

Does this also work with s3n paths?

Erik

[Sam Ritchie]

Yup, just make sure the path is prefixed with s3n://.

[sourab...@corp.247customer.com]

Any idea how can I use source-pattern in jcascalog?
In we can pass it in cascalog like: (hfs-textline "/root/path" :source-pattern "/2011/*/01/*")

In jcascalog we use Api.hfsTextline(path). Is there any way to pass the pattern as argument to Api.hfsTextline instead of complete path.

Thanks
Sourabh

[David Kincaid]

I don't see it in Api.java, but I think you could pretty easily create it yourself. Something like this:

public static Object hfsTextline(String path, String pattern) {
        return Util.bootSimpleFn("cascalog.api", "hfs-textline").invoke(path, Keyword.intern("source-pattern"), pattern);
    }

(Util is cascalog.Util and Keyword is clojure.lang.Keyword)

That is completely untested, but that's the general idea I think.

[Revanth Revoori]

Hi David,

I have a similar problem,I have to include all the folders with a pattern 032014/7 , 042014/7 , 052014/7 in to a tap so that it will take all the files inside those folder.I also have to include specific Pail Structure,Please give me any help regarding this.

[Andrés Corrada-Emmanuel]

For the benefit of others, I'll share my hack to do arbitrary collections of files that may not obey a simple pattern. The key is the line that maps the :sink keyword to hfs-textline sequences.


(defn make-hours-multi-tap
  "Makes a single data tap out of multiple calendar hour subdirectories."
  [base-path start-hour end-hour path-pattern]
  (let
    [hour-paths (t-paths/make-calendar-hour-paths base-path start-hour end-hour)
     ;; Not all hour paths may be present so we filter out the ones that are non-existent or empty
     hour-paths-with-files (filter hour-path-has-files? hour-paths)]
    (if (empty? hour-paths-with-files)
      nil
      ;; There are files with data that we can tap
      (let [hour-taps (map #(:sink (hfs-textline (str %  path-pattern))) hour-paths-with-files)]
        (MultiSourceTap. (into-array cascading.tap.Tap hour-taps))))))

[Beau Fabry]

Thanks for sharing this Andrés! We were hacking something up with globbing prior to my seeing this.

[Matthew Wooller]

Thank you - I had been trying to pull something together for a couple of hours, then found this - and as usual, I was overcomplicating things - this is nice and straight forward...

Cheers!

M.