dumping temporary values between subqueries
Andrew Xue
I have a query which uses the same subquery several times. It looks
something like
S3 SrcData => QueryA => QueryB => QueryC => etc.
^
||
S3 SrcData => QueryA
QueryA is being called twice and it would be nice to be able to
"cache" the result in a temp file in HDFS instead of running the query
twice. This is especially true because QueryA is filter job on the
SrcData and the cached result would be much smaller.
I found a few threads in the cascading list which says to implement
the isSafe() function on Operation to return false
http://groups.google.com/group/cascading-user/browse_thread/thread/59e3463093c1eebb#
http://groups.google.com/group/cascading-user/browse_thread/thread/cd283dadc6f76bbe/d09111e95b6a8852?lnk=gst&q=%22Dumping+pipe+to+disk#d09111e95b6a8852
Is there anyway to access and implement from Cascalog? Thanks
Andy
Andrew Xue
https://docs.google.com/drawings/d/1JClgjaMOimujRtOThMoSZCihcvG6k_8FWdkAe-QS7NU/edit
On Dec 16, 1:07 pm, Andrew Xue <and...@lumoslabs.com> wrote:
> Hi
>
> I have a query which uses the same subquery several times. It looks
> something like
>
> S3 SrcData => QueryA => QueryB => QueryC => etc.
> ^
> ||
> S3 SrcData => QueryA
>
> QueryA is being called twice and it would be nice to be able to
> "cache" the result in a temp file in HDFS instead of running the query
> twice. This is especially true because QueryA is filter job on the
> SrcData and the cached result would be much smaller.
>
> I found a few threads in the cascading list which says to implement
> the isSafe() function on Operation to return false
>
nathanmarz
a reduce step... if it's map-only (e.g. just a filter) it will redo
query A. I opened up an issue to expose the isSafe method for
subqueries so you can force this optimization: https://github.com/nathanmarz/cascalog/issues/38
You can probably force this optimization now by implementing a regular
Cascading filter that always returns false and sets that isSafe
method. So something like:
(<- [?foo ?bar] (source ?foo ?bar) (my-filter ?foo)
((IdentityUnsafe.) ?foo) (:distinct false))
where IdentityUnsafe is your Cascading filter implementation.
On Dec 16, 1:11 pm, Andrew Xue <and...@lumoslabs.com> wrote:
> ok, the ascii picture didn't work so here is a google doc drawing
>
> On Dec 16, 1:07 pm, Andrew Xue <and...@lumoslabs.com> wrote:
>
>
>
>
>
>
>
> > Hi
>
> > I have a query which uses the same subquery several times. It looks
> > something like
>
> > S3 SrcData => QueryA => QueryB => QueryC => etc.
> > ^
> > ||
> > S3 SrcData => QueryA
>
> > QueryA is being called twice and it would be nice to be able to
> > "cache" the result in a temp file in HDFS instead of running the query
> > twice. This is especially true because QueryA is filter job on the
> > SrcData and the cached result would be much smaller.
>
> > I found a few threads in the cascading list which says to implement
> > the isSafe() function on Operation to return false
>
Sam Ritchie
(workflow ["/tmp/checkpoints"]
queryA ([:tmp-dirs query-a-data]
(?- (hfs-seqfile query-a-data)
(query-a s3-path)))
queryB ([:deps queryA :tmp-dirs query-b-data]
(?- (hfs-seqfile query-b-data)
(query-b query-a-data)))
queryC ([:deps [queryA queryB] :tmp-dirs query-c-data]
(?- (hfs-seqfile query-c-data)
(query-c query-a-data query-b-data))))
and so on and so forth.
Sam Ritchie, Twitter Inc
703.662.1337
@sritchie09
(Too brief? Here's why! http://emailcharter.org)
--
Andrew Xue
checkpoint working -- have you seen this error before?
ClassCastException java.lang.String cannot be cast to
clojure.lang.IFn cascalog.contrib.checkpoint/exec-workflow!/
iter--210--214/fn--215 (checkpoint.clj:88)
i get this error with the workflow i was testing as well as when i cut
and pasted in the checkpoint_test.clj code and tried to (run-test!)
On Dec 21, 6:56 pm, Sam Ritchie <sritchi...@gmail.com> wrote:
> Hey, you'll probably find cascalog.checkpoint in cascalog-contrib helpful.
> I discuss an example here:http://sritchie.github.com/2011/11/15/introducing-cascalogcontrib.html.
> Something like
> (workflow ["/tmp/checkpoints"]
> queryA ([:tmp-dirs query-a-data]
> (?- (hfs-seqfile query-a-data)
> (query-a s3-path)))
> queryB ([:deps queryA :tmp-dirs query-b-data]
> (?- (hfs-seqfile query-b-data)
> (query-b query-a-data)))
> queryC ([:deps [queryA queryB] :tmp-dirs query-c-data]
> (?- (hfs-seqfile query-c-data)
> (query-c query-a-data query-b-data))))
> and so on and so forth.
>
> On Fri, Dec 16, 2011 at 1:11 PM, Andrew Xue <and...@lumoslabs.com> wrote:
>
> > ok, the ascii picture didn't work so here is a google doc drawing
>
>
>
>
>
>
>
>
>
> > On Dec 16, 1:07 pm, Andrew Xue <and...@lumoslabs.com> wrote:
> > > Hi
>
> > > I have a query which uses the same subquery several times. It looks
> > > something like
>
> > > S3 SrcData => QueryA => QueryB => QueryC => etc.
> > > ^
> > > ||
> > > S3 SrcData => QueryA
>
> > > QueryA is being called twice and it would be nice to be able to
> > > "cache" the result in a temp file in HDFS instead of running the query
> > > twice. This is especially true because QueryA is filter job on the
> > > SrcData and the cached result would be much smaller.
>
> > > I found a few threads in the cascading list which says to implement
> > > the isSafe() function on Operation to return false
>
> ..
>
>
> > > Is there anyway to access and implement from Cascalog? Thanks
>
> > > Andy
>
> --
> Sam Ritchie, Twitter Inc
> 703.662.1337
> @sritchie09
Sam Ritchie
[cascalog-checkpoint "0.1.0"]
instead of the global cascalog-contrib; I changed the blog post, but I haven't been able to figure out how to take the cascalog-contrib jar off of clojars.
Andrew Xue
On Dec 25, 5:54 pm, Sam Ritchie <sritchi...@gmail.com> wrote:
> Andy, try usinghttp://clojars.org/cascalog-checkpoint, or
Andrew Xue
for example something like:
(defn inner-workflow
[input-path output-path]
(workflow ["tmp/inner-workflow"]
step 1 ...
step 2 ... etc
))
(defn outer-workflow
[input-path output-path]
(workflow ["tmp/outer-workflow"]
step 1 ([:tmp-dirs inner-workflow-staging] (inner-workflow input-
path inner-workflow-staging))
step 2 ... etc
))
nathanmarz
to delete "/tmp/outer-workflow" but not "/tmp/inner-workflow", you'd
run into problems. If the workflow had previously only run part way
through inner-workflow, inner-workflow will end up emitting stale
results if outer-workflow is rerun.
Andrew Xue
collectify
i am using cacsalog-1.8.5-SNAPSHOT and util.clj no longer has this
function