Peon Port Assignment?

[Brian O'Neill]

Sorry, my google foo is failing me...

Does anyone know how peon port assignment works?

I see that you can set the start port, but it is unclear to me how that relates to port assignment.

Specifically, we are running multiple middle managers.  On one of those middle managers, I see the following port assignments over time:

zgrep Announcing *.log* | grep port | grep firehose | awk '{print $1, $9}'

druid-02-16-2016-1.log.gz:18:44:48.725 port=8105}]

druid-02-16-2016-2.log.gz:19:09:29.551 port=8106}]

druid-02-16-2016-3.log.gz:21:07:59.074 port=8107}]

druid-02-16-2016-4.log.gz:21:09:17.632 port=8108}]

druid-02-17-2016-1.log.gz:16:09:26.897 port=8110}]

druid-02-17-2016-2.log.gz:18:30:22.737 port=8111}]

druid-02-17-2016-3.log.gz:18:30:42.937 port=8112}]

druid-02-17-2016-3.log.gz:18:30:46.641 port=8113}]

druid-02-17-2016-3.log.gz:18:30:48.366 port=8114}]

druid-02-18-2016-2.log.gz:00:00:27.369 port=8124}]

druid-02-18-2016-6.log.gz:01:00:17.154 port=8125}]

druid-02-18-2016-7.log.gz:02:00:16.839 port=8126}]

druid-02-20-2016-2.log.gz:01:15:56.776 port=8148}]

druid-02-20-2016-3.log.gz:03:05:21.621 port=8149}]

druid-02-22-2016-1.log.gz:00:00:16.920 port=8160}]

druid-02-22-2016-6.log.gz:01:00:16.954 port=8161}]

druid-02-22-2016-7.log.gz:02:00:17.052 port=8162}]

druid-02-23-2016-1.log.gz:00:00:17.238 port=8185}]

druid-02-23-2016-6.log.gz:01:00:27.687 port=8186}]

druid-02-23-2016-7.log.gz:02:01:04.000 port=8187}]

druid-02-24-2016-1.log.gz:14:00:17.114 port=8199}]

druid-02-24-2016-2.log.gz:15:48:16.067 port=8200}]

druid-02-24-2016-3.log.gz:16:11:18.870 port=8201}]

druid-02-24-2016-4.log.gz:17:00:16.798 port=8202}]

druid-02-24-2016-5.log.gz:18:00:21.110 port=8203}]

druid-02-24-2016-6.log.gz:19:44:16.208 port=8204}]

druid-02-24-2016-7.log.gz:20:00:17.356 port=8205}]

druid.log:21:02:17.175 port=8206}]

How high will those ports go?  In this environment, we are running with 24 slots.

(We are trying to determine what port range to open for peon access)

thanks in advance,

-brian

-- 
Brian O'Neill
Principal Architect @ Monetate
m: 215.588.6024
bon...@monetate.com

Watch our Video | Marketing Blog | Free Resources | Follow Monetate on Twitter | Download our Resources App 

Know a great software engineer and want a referral bonus even if YOU don't work at Monetate? Tell our recruiters about them!

[Gian Merlino]

Hey Brian,

The port assignment is implemented here. Normally "findUnusedPort" is the one that gets called. https://github.com/druid-io/druid/blob/druid-0.8.3/indexing-service/src/main/java/io/druid/indexing/overlord/PortFinder.java

Note that in older versions of Druid the port assignment was done differently and the ports could potentially grow a lot higher. I think by 0.7.0 this new way was being used.

Gian

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/CAJHHpg33e%2BKX3Gz4fxr%3DeKWTMDiG3O2Qty%2BMpvH_%3Du9yBCab3g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Brian O'Neill]

Giam,

Thanks.  I don't see anything that would limit the port range.  The PortFinder appears to be used by the ForkingTaskRunner, which only returns childChatHandlerPorts to the pool (via the markPortUnused method).    Can you help me understand where/how the childPorts are re-used?

Or should we just assume that there is no finite range? (and open up all ports in the security group?)

-brian

[Brian O'Neill]

(and oops Gian != Giam, mea culpa)

On Wednesday, 24 February 2016 16:27:47 UTC-5, Brian O'Neill wrote:

[charles.allen]

thanks for pointing this out, it is a bug that I verified in our prod environment (port numbers are getting higher, but not high enough to be a concern yet), can you file it at https://github.com/druid-io/druid/issues ?

If tasks run on every hour or so the length of time before you run out of ports is measured in a few years, depending on how much you pack your nodes. But it shouldn't be a failure mode, and the port ranges used should be predictable.

[Brian O'Neill]

Absolutely.  Will do.

-- 
Brian O'Neill
Principal Architect @ Monetate
m: 215.588.6024
bon...@monetate.com

Watch our Video | Marketing Blog | Free Resources | Follow Monetate on Twitter | Download our Resources App 

Know a great software engineer and want a referral bonus even if YOU don't work at Monetate? Tell our recruiters about them!

[Brian O'Neill]

https://github.com/druid-io/druid/issues/2545

-- 
Brian O'Neill
Principal Architect @ Monetate
m: 215.588.6024
bon...@monetate.com

Watch our Video | Marketing Blog | Free Resources | Follow Monetate on Twitter | Download our Resources App 

Know a great software engineer and want a referral bonus even if YOU don't work at Monetate? Tell our recruiters about them!

On Feb 25, 2016, at 10:44 AM, charles.allen <charle...@metamarkets.com> wrote: