"Too many counters"

[PaulON]

Hey all,

we keep running into this issue and are kinda stuck on it!

It seems that no matter what we set in the configs, our jobs that are launched via oozie randomly throw this error 

"org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many counters: 121 max=120"

We are using Cascading 3.2.1 and have tried overriding the default for mapreduce.job.counters.max everywhere....but it seems like the Limits class is sometimes being inited with a null config..

We'd rather not just remove the counters but thats our next step...

Hoping someone can provide some insight or ideas to try!

Cheers,

Paul

[Chris K Wensel]

This sounds like a Hadoop property setting…

A quick search uncovered this

https://stackoverflow.com/questions/20899050/org-apache-hadoop-mapreduce-counters-limitexceededexception-too-many-counters

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/2ab959f6-a251-417b-9fe3-7388b9d4a189%40googlegroups.com.

—

Chris K Wensel

ch...@wensel.net

[PaulON]

Yeah, its is a Hadoop setting (used in org.apache.hadoop.mapreduce.counters.Limits), 

public static synchronized void init(Configuration conf) {
    if (!isInited) {
        if (conf == null) {
            conf = new JobConf();
        }

        COUNTERS_MAX = ((Configuration)conf).getInt("mapreduce.job.counters.max", 120);

The problem is that we are seeing inconsistent behaviour which based on the above code seems to stem from when/how this class is initialised.

*sometimes* we get an error that we have exceeded the 120, despite us overriding it in the config, othertimes it runs through just fine (with our 500+ counters)

I'm not able to trace how this class is being initialised via Cascading, was hoping someone could share some info so we could maybe see if there is some race condition or some other way/place we can set the override.

On Monday, July 29, 2019 at 3:10:16 PM UTC+1, Chris K Wensel wrote:

This sounds like a Hadoop property setting…

A quick search uncovered this

https://stackoverflow.com/questions/20899050/org-apache-hadoop-mapreduce-counters-limitexceededexception-too-many-counters

On Jul 29, 2019, at 2:21 AM, PaulON <pone...@gmail.com> wrote:

Hey all,

we keep running into this issue and are kinda stuck on it!

It seems that no matter what we set in the configs, our jobs that are launched via oozie randomly throw this error 

"org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many counters: 121 max=120"

We are using Cascading 3.2.1 and have tried overriding the default for mapreduce.job.counters.max everywhere....but it seems like the Limits class is sometimes being inited with a null config..

We'd rather not just remove the counters but thats our next step...

Hoping someone can provide some insight or ideas to try!

Cheers,

Paul

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cascadi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/2ab959f6-a251-417b-9fe3-7388b9d4a189%40googlegroups.com.

—

Chris K Wensel

ch...@wensel.net

[Chris Schneider]

Hi Paul,

I don’t think Cascading initializes this class. I would instead investigate how you’re overriding the property. The best method is to override it in your mapred-site.xml, and to ensure that this same file is placed within the ./hadoop/conf/ folder on each node of your Hadoop cluster.

Good Luck,

- Chris

To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/95b3325d-9420-448b-9b3f-f15f9577151b%40googlegroups.com.

-----------------------------------------
Chris Schneider
http://www.scaleunlimited.com
custom big data solutions
-----------------------------------------

[Ben Podgursky]

I vaguely recall (could very well be wrong, long time ago) that you have to set this on the NodeManagers themselves.  Setting it per-job lets it aggregate more than 120 at the AM level (if the unique counters are partitioned across tasks), but you'll still fail I think if an individual task goes over 120.

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/1D429DE9-5982-48DA-9024-90353DE81F91%40scaleunlimited.com.