ERROR 1200: could not instantiate 'pigpen.PigPenFnAlgebraic'

[Punit Naik]

I have attached my pig script below which I have generated from the code

(require '[pigpen.core :as pig])

(require '[pigpen.pig :as ws])

(require '[pigpen.fold :as fold])

(defn data []

(let [x (pig/load-json "/pigpen-test")

      grouped (->> (pig/group-by :ga_pagePath {:fold (->> (fold/map :ga_dateHour) (fold/max))} x) (pig/map (fn [[pagePath max_dateHour]] {:ga_pagePath pagePath :ga_dateHour_max max_dateHour})))]

(do

(let [v (pig/join [(grouped :on :ga_pagePath) (x :on :ga_pagePath)] (fn [x y] (conj x y)))]

(->>

(pig/group-by :ga_pagePath {:fold (->> (fold/filter (fn [z] (<= (Math/abs (- (read-string (:ga_dateHour z)) (read-string (:ga_dateHour_max z)))) 1))) (fold/map (fn [q] (read-string (read-string (reduce read-string (vals (get-in q [:ga_pageviews]))))))) (fold/sum))} v)

(pig/map (fn [[pagePath ga_total_pageviews]] {:ga_pagePath pagePath :ga_total_pageviews_in_last_hour ga_total_pageviews}))

(pig/store-json "/pigpen-test-out")

)

)

)

)

)

And this is the stacktrace:

2016-03-11 15:02:25,163 [main] ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. could not instantiate 'pigpen.PigPenFnAlgebraic' with arguments '[, (->> (fold/map :ga_dateHour) (fold/max))]'

Failed to parse: could not instantiate 'pigpen.PigPenFnAlgebraic' with arguments '[, (->> (fold/map :ga_dateHour) (fold/max))]'

at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:201)

at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1735)

at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1443)

at org.apache.pig.PigServer.parseAndBuild(PigServer.java:387)

at org.apache.pig.PigServer.executeBatch(PigServer.java:412)

at org.apache.pig.PigServer.executeBatch(PigServer.java:398)

at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171)

at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)

at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)

at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)

at org.apache.pig.Main.run(Main.java:502)

at org.apache.pig.Main.main(Main.java:177)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Caused by: java.lang.RuntimeException: could not instantiate 'pigpen.PigPenFnAlgebraic' with arguments '[, (->> (fold/map :ga_dateHour) (fold/max))]'

at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:772)

at org.apache.pig.newplan.logical.expression.UserFuncExpression.getFieldSchema(UserFuncExpression.java:222)

at org.apache.pig.newplan.logical.optimizer.FieldSchemaResetter.execute(SchemaResetter.java:264)

at org.apache.pig.newplan.logical.expression.AllSameExpressionVisitor.visit(AllSameExpressionVisitor.java:143)

at org.apache.pig.newplan.logical.expression.UserFuncExpression.accept(UserFuncExpression.java:113)

at org.apache.pig.newplan.ReverseDependencyOrderWalker.walk(ReverseDependencyOrderWalker.java:70)

at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)

at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visitAll(SchemaResetter.java:67)

at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:122)

at org.apache.pig.newplan.logical.relational.LOGenerate.accept(LOGenerate.java:245)

at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)

at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:114)

at org.apache.pig.parser.LogicalPlanBuilder.buildForeachOp(LogicalPlanBuilder.java:1055)

at org.apache.pig.parser.LogicalPlanGenerator.foreach_clause(LogicalPlanGenerator.java:15896)

at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1933)

at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)

at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)

at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)

at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191)

... 16 more

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:740)

... 34 more

Caused by: java.lang.RuntimeException: Exception evaluating: (->> (fold/map :ga_dateHour) (fold/max))

at pigpen.runtime$eval_string.invoke(runtime.clj:184)

at clojure.lang.AFn.applyToHelper(AFn.java:154)

at clojure.lang.AFn.applyTo(AFn.java:144)

at clojure.core$apply.invoke(core.clj:624)

at clojure.core$memoize$fn__5097.doInvoke(core.clj:5846)

at clojure.lang.RestFn.invoke(RestFn.java:408)

at clojure.lang.Var.invoke(Var.java:379)

at pigpen.PigPenFnAlgebraic.<init>(PigPenFnAlgebraic.java:62)

... 39 more

Caused by: java.lang.RuntimeException: No such namespace: fold, compiling:(NO_SOURCE_PATH:0:0)

at clojure.lang.Compiler.analyze(Compiler.java:6464)

at clojure.lang.Compiler.analyze(Compiler.java:6406)

at clojure.lang.Compiler$InvokeExpr.parse(Compiler.java:3665)

at clojure.lang.Compiler.analyzeSeq(Compiler.java:6646)

at clojure.lang.Compiler.analyze(Compiler.java:6445)

at clojure.lang.Compiler.analyze(Compiler.java:6406)

at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5782)

at clojure.lang.Compiler$FnMethod.parse(Compiler.java:5217)

at clojure.lang.Compiler$FnExpr.parse(Compiler.java:3846)

at clojure.lang.Compiler.analyzeSeq(Compiler.java:6642)

at clojure.lang.Compiler.analyze(Compiler.java:6445)

at clojure.lang.Compiler.eval(Compiler.java:6700)

at clojure.lang.Compiler.eval(Compiler.java:6666)

at clojure.core$eval.invoke(core.clj:2927)

at pigpen.runtime$eval_string.invoke(runtime.clj:182)

... 46 more

Caused by: java.lang.RuntimeException: No such namespace: fold

at clojure.lang.Util.runtimeException(Util.java:221)

at clojure.lang.Compiler.resolveIn(Compiler.java:6910)

at clojure.lang.Compiler.resolve(Compiler.java:6884)

at clojure.lang.Compiler.analyzeSymbol(Compiler.java:6845)

at clojure.lang.Compiler.analyze(Compiler.java:6427)

... 60 more

What is the problem?

[Matt Bossenbroek]

You need to put the code in an actual namespace that gets compiled & shipped with the script as an uberjar. Instructions for building the uberjar are at the end of the tutorial [1].

Writing scripts in a REPL will work for very basic examples, but not much else.

-Matt

[1] https://github.com/Netflix/PigPen/wiki/Tutorial

--
You received this message because you are subscribed to the Google Groups "PigPen Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pigpen-suppor...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Attachments:

- final.pig

[Punit Naik]

Hello Matt

I put my code in the core.clj file of my project, created the script and the uberjar and then again ran the program. Still it show me the same error.

[Matt Bossenbroek]

Could you send me the jar & script (off-thread) and I'll take a look?

-Matt

[Matt Bossenbroek]

In case anyone finds this thread in the future, the issue was that Punit's code was in the `user` namespace. For pigpen to work properly, your code must be in a real file with a real namespace that gets packaged into an uberjar and shipped along with the generated script.

-Matt

[Punit Naik]

Okay. 

I would like to rectify my solution though.

If I set the core.clj file as main in the project.clj, it fails to create an uberjar. So you have to remove ':main' from project.clj.

So the correct solution is to have another function in the file where your code resides and that function should use the 'pigpen.pig/write-script' function to write the script.

After doing that you should go in the project directory and create an uberjar first.

Then do a 'lein repl' while being in the project directory.

Now that you are in the 'user' namespace, do a '(use '[pigpen-demo.core] :reload)' which will load the core.clj file.

NOTE: My project name is 'pigpen-demo'

Now you have to call that function which writes the script and you are done.

use that script with pig to execute your job.