Storm Supervisor MaxResourceCapability

[Roshan Punnoose]

I recently tried to run storm-yarn on my cluster, and noticed that the supervisor jobs were asking YARN for a very large amount of memory. Looking at the code, it seems that the supervisor is asking for the MaxResourceCapability defined by the ResourceManager. Is this correct? For some reason, my YARN RM kept saying that it could not satisfy a container of this size. I changed the code for a quick test to use the master.container-size-mb property that is really only for the ApplicationMaster memory size, and it worked well. The supervisors came up without any trouble.

Also, where do the worker logs go to? I can't seem to find it on disk. (I found the supervisor.log, but not the worker-*.log)

One more question :) the worker jobs come up as separate applications, not as YARN resources. Does that mean if I set the worker to -Xmx10G, I need to make sure that the container running the supervisor is set to have 40G+ resource capability in YARN? (10G x possible 4 workers)

Thanks again! This is a very cool project. 

Roshan

[Bobby Evans]

Sorry I didn't respond sooner.  Life has been rather crazy.  This is an ugly hack to try to work around storm not supporting ephemeral ports on the workers, and YARN not able to restrict containers to run on different nodes.  I really would like to rip it out but without it workers will start to have collisions if two supervisors are run on the same physical node.

Hopefully in the not too distant future we can remove this.

--Bobby

--
You received this message because you are subscribed to the Google Groups "storm-yarn" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-yarn+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Roshan Punnoose]

That makes perfect sense. What would you recommend if I want to have other yarn applications coexisting with Storm? We are trying to run Hoya (Accumulo/Hbase) as well, and want them to be able to exist on the same nodes. 

[Bobby Evans]

If you feel like coding you could try to modify the AM to return containers that are on nodes that already have a supervisor running on them.  Then you could change the request size to be configurable instead of going off of the  maximum request. 

--Bobby 

On my phone