filter fn causing a failure. I need some eyes.
52 views
Skip to first unread message
Eric Gebhart
Jun 9, 2016, 5:24:41 PM6/9/16
to cascalog-user
hello everyone,
I just can't see what is wrong here. It looks ok to me. And the filter function works fine if
I use it directly with filter, but I get a very short traceback using it in a query.
This is basically a category vector, I just want to filter based on if the vectors match
always matching for the length of the short one, which will always be in the specified list
not the other way around. I've written lots of filter functions but I think I must be going blind.
Maybe all I need is some fresh eyes to take a look.
In addition to the working code below, I also tried using "cats" as a sub-query
with (<-....) so that fcats looked like this:
(defn cats []
(<- [?id ?catv]
(cat-tap ?id ?lvl ?code)
(:sort ?lvl)
(mk-vec ?code :> ?catv)))
(defn fcats [catf]
(let [prods (cats)]
(??<- [?id ?catv]
(cats ?id ?catv)
;(catf ?catv)
)))
This has been a mixed bag. It makes me suspect my environment.
I've seen it work, and and I've seen unhandled exceptions, and also this error which is
better than nothing. Although I can't seem to get this error back.
Tails required in merge-tails. (not-empty tails)
At the moment, using a sub-query is not working at all for this example. I think it would
simplify the problem a bit if it did.
I can't say how many times I've written queries like that. What is it about this one?
I have noticed that when I use real code, I write the sub query usage like this: (cats :> ?id ?catv)
which works fine for my more complex production code that this example is modeled after.
The other thing I tried was turning my filter-fn into map fn so I could return the result
and see it in the output. That actually worked for a while. Then I restarted my REPL and it
stopped. Again, I'm suspicious something weird is going on.
Now, it results in the same crash as the filter does.
(defn catv-filter
"Returns a function that filters by none, or by a lists of category vectors.
Takes an empty list or a list of vectors of category numbers. The filter takes a
vector of category numbers. The vector will be shortened to match the comparison
category vectors as needed so that sub categories will match. "
[categories]
(filterfn [v]
(if (empty? categories)
true
(reduce #(or (= (vec (take (count %2) v)) %2) %1) false categories))))
(defmapfn catv-filter-res
"Returns a function that filters by none, or by a lists of category vectors.
Takes an empty list or a list of vectors of category numbers. The filter takes a
vector of category numbers. The vector will be shortened to match the comparison
category vectors as needed so that sub categories will match. "
[categories]
(fn [v]
(if (empty? categories)
true
(reduce #(or (= (vec (take (count %2) v)) %2) %1) false categories))))
(defbufferfn mk-vec [tuples]
[[(reduce #(conj %1 %2) [] (map first tuples))]])
(def cat-tap [["foo" 1 3]
["foo" 2 100]
["foo" 3 8]
["foo" 4 12]
["bar" 1 5]
["bar" 2 22]
["bar" 3 36]
["bar" 4 212]
["baz" 1 3]
["baz" 2 100]
["baz" 3 8]
["baz" 4 300]])
(def allcats (catv-filter nil))
(def catf (catv-filter '([3 100 8 12])))
(def catf2 (catv-filter '([3 100 8])))
(defn cats []
(??<- [?id ?catv]
(cat-tap ?id ?lvl ?code)
(:sort ?lvl)
(mk-vec ?code :> ?catv)))
(defn catsres [s]
(let [fres (catv-filter-res s)]
(??<- [?id ?catv ?res]
(cat-tap ?id ?lvl ?code)
(:sort ?lvl)
(mk-vec ?code :> ?catv)
(fres ?catv :> ?res))))
(defn fcats [catf]
(??<- [?id ?catv]
(cat-tap ?id ?lvl ?code)
(:sort ?lvl)
(mk-vec ?code :> ?catv)
(catf ?catv)))
(defn filter-test [f]
(filter #(f (second %)) (cats)))
test> (cats)
(["bar" [5 22 36 212]] ["baz" [3 100 8 300]] ["foo" [3 100 8 12]])
test> (filter-test allcats)
(["bar" [5 22 36 212]] ["baz" [3 100 8 300]] ["foo" [3 100 8 12]])
test> (filter-test catf)
(["foo" [3 100 8 12]])
test> (filter-test catf2)
(["baz" [3 100 8 300]] ["foo" [3 100 8 12]])
--test> (fcats allcats)
Unhandled cascading.flow.FlowException
local step failed
FlowStepJob.java: 219 cascading.flow.planner.FlowStepJob/blockOnJob
FlowStepJob.java: 149 cascading.flow.planner.FlowStepJob/start
FlowStepJob.java: 124 cascading.flow.planner.FlowStepJob/call
FlowStepJob.java: 43 cascading.flow.planner.FlowStepJob/call
FutureTask.java: 266 java.util.concurrent.FutureTask/run
ThreadPoolExecutor.java: 1142 java.util.concurrent.ThreadPoolExecutor/runWorker
ThreadPoolExecutor.java: 617 java.util.concurrent.ThreadPoolExecutor$Worker/run
Thread.java: 745 java.lang.Thread/run
Eric Gebhart
Jun 13, 2016, 2:12:19 PM6/13/16
to cascalog-user
Maybe someone can see something I don't see here, but otherwise it looks to me like anonymous filter functions are broken.
I have a very simple example here that shows the problem clearly.
I have a very simple example here that shows the problem clearly.
An anonymous filter function doesn't work. But the same function used directly does.
This is happening in cascalog 3.0.0.
This is happening in cascalog 3.0.0.
(defn lvlf [max lvl] (< lvl max))
(defn lvl-f
[max]
(fn [lvl] (lvlf max lvl)))
(defn codes [max]
(let [lvl-filter (lvl-f max)]
(??<- [?id ?lvl ?code]
(cat-tap ?id ?lvl ?code)
(:sort ?lvl)
;(< ?lvl max) ; works.
;(lvlf max ?lvl) ; works
; (lvl-filter ?lvl) ; exception.
)))
(def cat-tap [["foo" 1 3]
["foo" 2 100]
["foo" 3 8]
["foo" 4 12]
["bar" 1 5]
["bar" 2 22]
["bar" 3 36]
["bar" 4 212]
["baz" 1 3]
["baz" 2 100]
["baz" 3 8]
["baz" 4 300]])
test> (codes 3) ;; with (< max ?lvl)
(["foo" 1 3] ["foo" 2 100] ["bar" 1 5] ["bar" 2 22] ["baz" 1 3] ["baz" 2 100])
test> (codes 3) ;; with (lvlf max ?lvl
(["foo" 1 3] ["foo" 2 100] ["bar" 1 5] ["bar" 2 22] ["baz" 1 3] ["baz" 2 100])
test> (codes 3) ;; with (lvl-filter ?lvl)
FlowException local step failed cascading.flow.planner.FlowStepJob.blockOnJob (FlowStepJob.java:219)
Sam Ritchie
Jun 13, 2016, 3:21:53 PM6/13/16
to cascal...@googlegroups.com
Oh, yeah, you definitely have to use the anonymous function macro that Cascalog supplies in `cascalog.logic.fn`. You can also just use `filterfn` in cascalog.api and everything will work great:
(defn lvl-f
[max]
(filterfn [lvl] (lvlf max lvl)))
--
You received this message because you are subscribed to the Google Groups "cascalog-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascalog-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Eric Gebhart
Jun 13, 2016, 5:48:25 PM6/13/16
to cascalog-user
Thanks Sam.
I must have changed something else also. I have always used filterfn but changed it today to see if it made a difference after this simple example failed.
Oddly enough it worked right after the change to filterfn, but failed on a second run immediately after the first. Which is what I have been seeing if it works at all.
Other than a bit of refactoring what I have now is pretty much what I had in my first post.
After it failed the second time I reloaded the entire file, and now it seems to be working consistently.
It has been behaving very strangely. I've written bunches of filter functions like these and never had any problems.
one unrelated edit, a reload,
And now it fails again, continuously. I think I'll reboot.
Eric Gebhart
Jun 15, 2016, 5:03:09 PM6/15/16
to cascalog-user
I'm still seeing the same problems for no apparent reason.
With a fresh REPL the code fails with a stacktrace.
Reloading the code doesn't always make it start working but sometimes does.
Sometimes just loading a defun will fix it also.
Editing something and loading just the change might also cause it to start failing.
This is really weird behavior. With all this trouble I thought I should update my environment.
I was using cider 10, so its fairly old, but has been very stable up until now.
It took almost a day, but I have everything working again with cider 12.
Still this problem persists. Perfectly good code gives stack traces.
And then starts working.
I just did a simple test. It was working.
I deleted an unrelated function and reloaded all the code.
It then started crashing. I reloaded the code again and it started working.
It's not cascalog, I don't believe it is cider or emacs, it seems more like there is something wrong lower down.
Occasionally I get these errors, but not in the above case..
nrepl-send-sync-request: Sync nREPL request timed out
It's very strange. If anyone has any ideas I'd appreciate them.
Eric
Eric Gebhart
Jun 16, 2016, 6:58:00 PM6/16/16
to cascalog-user
I'm still working on this problem.
1. I created a simple project with just this code. --- It works great. :-/
2. The bad behavior is repeatable in my production code.
a. reloading the code after adding or deleting a blank line causes a stack trace from then on.
b. reloading twice in a row has no effect.
c. reloading a second time after a stacktrace causes the code to begin working.
3. there are no errors anywhere that I can find.
4. it works fine in lein repl - mostly. I have to explicitly load it.
user=> (require 'tools.products)
user=> (tools.products/codes 3)
(["foo" 1 3] ["foo" 2 100] ["bar" 1 5] ["bar" 2 22] ["baz" 1 3] ["baz" 2 100])
5. The code always has to be loaded explicitly before the symbols are resolved and it can crash the first time - in emacs, or work in lein repl.
It seems that there is an error somewhere, but I haven't found one yet.
Cider behaves less well than lein repl....
I haven't been able to replicate this problem in a simple way.
Probably, no one here is interested in this, Cascalog seems to be working fine although interacting badly with something I have not identified.
Sam Ritchie
Jun 16, 2016, 7:12:10 PM6/16/16
to cascal...@googlegroups.com
Hey Eric! Sorry for not giving any input on this. I'm always suspicious of AOT compilation - but I'm far enough away from Cascalog in my day-to-day that my intuition here is not very good.
I think it's great that you're sending the reports to the list, and hope that we can find a solution, or that anyone with any idea of what's going on will send in some input.
--
You received this message because you are subscribed to the Google Groups "cascalog-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascalog-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Eric Gebhart
Aug 9, 2016, 2:37:05 PM8/9/16
to cascalog-user
Hey Sam, it's ok. I did figure it out. It was a couple of weeks of hair pulling. The problem is actually the use of clojure flycheck with eastwood. In emacs. Eastwood uses the current session for it's checks. It makes no guarantees that
it won't corrupt the session when used in this way. And that is what is happening. Reloading the entire source to refresh the session solves the problem. Or disabling Eastwood or clojure flycheck.
It's possible that clojure flycheck could use a secondary session for it's linting, especially since cider has made a second session pretty easy to get. But I haven't had the time to code that enhancement
and no one else seems to interested. The Eastwood developer is interested in finding the problem but is also time constrained. I have a project that duplicates the problem so I would guess that it
could get fixed someday.
Reply all
Reply to author
Forward
0 new messages