question about the PIG queue command

[Demidov, Igor V]

Hello,

I am working with pipen for couple weeks on a MapR Hadoop environment. I cannot run the pigpen in a mapreduce mode for some reasons but all the mocking in local mode works fine.

Do you know how to run PIG command like  

 SET mapred.job.queue.name 'coda_q1'  

Without this the MapR security doesn’t allow to run it.

When I generate the PIG script from my pigpen code and add this SET queue command everything works.

2) The second question is when I use DUMP command like (pipen/dump a) everything works (as it in local mode)

As soon as I replace it with something like (pigpen/store-csv) it stops working.

If you need more details I can send you additional info.

 

Best Regards

 

Igor Demidov | Optum Technology

Big Data Engineer, CODA

 

13625 Technology Drive Eden Prairie, MN 55344

W : 952-917-8533

igor.d...@optum.com

www.optum.com

 

 

 

 

This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.

[Matt Bossenbroek]

You can set pig options in a script using the following command: http://netflix.github.io/PigPen/pigpen.pig.html#var-set-options

There’s a slight difference in behavior between pig’s dump & store commands and pigpen’s. In pig, either will trigger an evaluation of the script. In pigpen, you can have multiple store commands but each will return only a query - you still need to call pig/dump to execute the query.

For example:

;; return data in the repl

(->> data

  (pig/dump))

;; write to a file & then return data in the repl

(->> data

  (pig/store-csv “out.csv")

  (pig/dump))

This is so you can do the following:

(->> data

  (pig/store-csv “out.csv")

  (pigpen.pig/write-script “script.pig"))

If the store command returned data, we wouldn’t be able to write the script.

HTH

-Matt

--
You received this message because you are subscribed to the Google Groups "PigPen Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pigpen-suppor...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.