Cascading counter

[Velkumar Neel]

Hi

If in a cascading Pipe as below,

Pipe P1 = new UniqueCount(P1,new Fields(Field1, Field2,Field3), new Fields(COUNT).

From the above pipe, I need to use the COUNT to get assigned to a local variable to display the statistics of how many records are present.

How should i assign this value?

(OR)

What is the other way to get the count value?

Thanks

Vel

[Ken Krugler]

---

From: Velkumar Neel

Sent: December 28, 2014 9:41:44pm PST

To: cascadi...@googlegroups.com

Subject: Cascading counter

You'll have N values for COUNT, one for each unique combination fo values for Field1, Field2, and Field3, right?

So it's not a single value, and it doesn't represent how many records were present, it represents how many groups you get from a key composed of those three fields.

In any case, some options...

1. Attach P1 to a text file sink Tap, then read that when the Flow has completed.

2. Attach P1 to a database sink Tap, and read those records when the Flow has completed.

3. If the number of unique combinations of fields is going to be "small" (say less than 100) then you could use a custom function that sythensizes a counter name from the field values and uses the count as the increment for that counter.

-- Ken

--------------------------

Ken Krugler

+1 530-210-6378

http://www.scaleunlimited.com

custom big data solutions & training

Hadoop, Cascading, Cassandra & Solr

[Velkumar Neel]

Hi ken

Thanks. But I cannot attach that pipe to tail sink because I have intermediate pipes whose count is also needed. So in this case what should I do?

Thanks

Vel

--
You received this message because you are subscribed to a topic in the Google Groups "cascading-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cascading-user/JtiSFDPNrpA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/8F70BB3B-0A4C-470B-9DEA-758E520B3A8B%40transpac.com.
For more options, visit https://groups.google.com/d/optout.

[Ken Krugler]

---

From: Velkumar Neel

Sent: December 29, 2014 12:06:55pm PST

To: cascadi...@googlegroups.com

Subject: Re: Cascading counter

Hi ken

Thanks. But I cannot attach that pipe to tail sink because I have intermediate pipes whose count is also needed. So in this case what should I do?

Split the pipe if you need it both as output and as input to subsequent Flow steps.

Pipe outputCountersPipe = new Pipe("output counters", P1);

Now continue using P1 in the rest of your Flow, and connect the outputCountersPipe to the sink.

-- Ken

You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.

To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/CAAZaL27WjS8EQTv9nZUWzQgu-DRfvabOPO4XHU%3DG2nmQX3Wm2g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Velkumar Neel]

Hi ken

Thanks. If we need to count how many records present in a pipe, how to do that?

On Monday, December 29, 2014, Ken Krugler <kkrugle...@transpac.com> wrote:

[Ken Krugler]

---

From: Velkumar Neel

Sent: December 30, 2014 3:10:22pm PST

To: cascadi...@googlegroups.com

Subject: Re: Cascading counter

Hi ken

Thanks. If we need to count how many records present in a pipe, how to do that?

http://docs.cascading.org/cascading/2.5/javadoc/cascading/operation/state/Counter.html

-- Ken

You received this message because you are subscribed to the Google Groups "cascading-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.

To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/CAAZaL27%2BcnajUqhr0aiLYJ21xd1-4guHLi3K93k8mc6WfRnKTg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Velkumar Neel]

Hi Ken

I have declared the counter as below 

Public enum counters

 public static Enum [] getCounter

{

return new Enum[]

{

Tot_no_of_records,

no_of_records_selected

}

In my cascading flow i have declared a counter as below

int j = 1;

Pipe = new Each(Pipe, new Counter(counters.getCounter, int J)

Iam getting counter value as zero. Am I doing anything wrong?

Thanks

Vel

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/CD15E05D-D77D-48A9-AB71-323E5703DA56%40transpac.com.

[Ken Krugler]

Hi Vel,

No idea what you're doing with the getCounter call.

Why not

public enum MyCounters {

Tot_no_of_records,

no_of_records,selected

}

And then

pipe = new Each(pipe, new Counter(MyCounters.Tot_no_of_records));

-- Ken

---

From: Velkumar Neel

Sent: January 6, 2015 1:01:40pm PST

[Velkumar Neel]

Hi Ken

Thanks. I try to display the counter value in local cascading, it didn't give the count value.

Sysprint (MyCounters.Tot_no_of_records).

--
You received this message because you are subscribed to a topic in the Google Groups "cascading-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cascading-user/JtiSFDPNrpA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/2784F557-2ADD-4D9F-B66F-08DB7E024AB2%40transpac.com.

[Velkumar Neel]

Hi ken 

Iam trying to print the pipe with the files values

But it is not showing in the debug logs. I worked on this product 5 years before . Please help me with

Iam saying 

Pipe = new Each(pipe , new fields (“name”),

New debug (“print”, true)

Is there any other setting I need to do ? 

Please note the pipe Iam debugging is an intermediate pipe .

Thanks in advance 

Sent from my iPhone

On Jan 7, 2015, at 11:05 AM, Velkumar Neel <neel...@gmail.com> wrote:

Hi Ken

[Ken Krugler]

When you run Cascading locally, I don’t think Hadoop counters work, because there is no real Job Manager.

Though it’s been a while (like almost 5 years) since I had to deal with that.

— Ken

You received this message because you are subscribed to the Google Groups "cascading-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/BC6C0A75-63EB-49A3-A59E-19F9A4428B1A%40gmail.com.

--------------------------

Ken Krugler

http://www.scaleunlimited.com

custom big data solutions & training

Hadoop, Cascading, Cassandra & Solr

--------------------------

Ken Krugler

[Velkumar Neel]

Hi Ken

Thanks for the prompt reply always 

Iam running on cluster. 

Sent from my iPhone

On Feb 8, 2021, at 5:50 PM, Ken Krugler <ken_...@krugler.org> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/14C64BBB-C7F5-4766-AA17-9506F31C659C%40krugler.org.

[Chris K Wensel]

The Debug filter will print to stdout wherever its run (it doesn’t rely on counters, fwiw). 

you might double check stdout isn’t redirected somewhere on your cluster non obvious.

Debug is also a PlannedOperation, and can be removed by the planner during planning. See what DebugLevel you set on the FlowDef. By default it is left in.

ckw

You received this message because you are subscribed to the Google Groups "cascading-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/BC6C0A75-63EB-49A3-A59E-19F9A4428B1A%40gmail.com.

[Velkumar Neel]

Does it need to be flowdef.verbose? It is now set to default . 

Sent from my iPhone

On Feb 8, 2021, at 6:30 PM, Chris K Wensel <ch...@wensel.net> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/cf97e319-ffab-4885-9e5c-a029e1ac901f%40Spark.

[Chris K Wensel]

The default is to leave in the Debug. 

Try running the flow locally to confirm it’s printing to stdout.

ckw

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/BC0CEB1A-C50E-4C64-A580-29301BD535EA%40gmail.com.

[Velkumar Neel]

Thank you both for the prompt reply .

It worked 

One more help please.

I have a source pipe which may or may not have records. In empty records case how do I handle the flowdef? 

Sent from my iPhone

On Feb 8, 2021, at 6:12 PM, Velkumar Neel <neel...@gmail.com> wrote:

Hi Ken

[Chris K Wensel]

I’m unsure what outcome you expect?

You can always test the sources for size before you plan the flow. Taps can be used independently.

Cascading 4 (under dev) https://github.com/cwensel/cascading has added new APIs to the Tap interface providing the Java stream API for reading data locally, for example.

ckw

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/5C6AB034-50F5-4F98-85F9-6869B4DA136A%40gmail.com.