Dynamic Categories, thread per category

[Chris Goffinet]

Hi

We've been experimenting with scribe and noticed an interesting situation. Lets say for example you have this:

<store>
category=Event*
type=network
remote_host=127.0.0.1
remote_port=1463
</store>

If I were to send messages with categories such as:

Event-1
Event-2
etc, it will create a thread per category seen. I was hoping that new_thread_per_category=false, would prevent this, but it does not.

Anthony what do you think about adding support to limiting the number of threads for categories that might be used in this way? I'd be happy to provide a patch if you think it makes sense.

-Chris

[Chris Goffinet]

So I dug more into this (source code). Looks like a documentation bug. I will go fix this.

new_thread_per_category: boolean

• If true, will create a new thread for every category seen. Otherwise, will only create a single thread for every store defined in the configuration.
• default true

The input takes, yes or no not boolean.

[Thomas A. McGonagle]

Nice Catch Chris,

 Did you dig into the source code? Where does scribe source and set its config file. I have had a couple of other similar experiences with the config file, it would be great to know where to check. 

-Tom

[anthony]

Chris, good catch. And thanks for updating the wiki.

Thomas, if you find any similar issues, please let me know!

-Anthony

[Chris Goffinet]

Yes I looked at the source code.

This helps:

http://github.com/facebook/scribe/blob/master/src/scribe_server.cpp#L571

I wonder if it might be good idea to consolidate everything to boolean.

[anthony]

I agree, it's definitely confusing having some params be yes/no while
others are true/false. Instead of switching to boolean, maybe we can
change all parameter to accept both yes/no and true/false. This way
everything will be less confusing and backwards compatible.

-Anthony

On Nov 12, 7:12 pm, Chris Goffinet <c...@chrisgoffinet.com> wrote:
> Yes I looked at the source code.
>
> This helps:
>
> http://github.com/facebook/scribe/blob/master/src/scribe_server.cpp#L571
>
> I wonder if it might be good idea to consolidate everything to boolean.
>

> On Thu, Nov 12, 2009 at 6:14 PM, Thomas A. McGonagle <mcgona...@gmail.com>wrote:
>
> > Nice Catch Chris,
> >  Did you dig into the source code? Where does scribe source and set its
> > config file. I have had a couple of other similar experiences with the
> > config file, it would be great to know where to check.
> > -Tom
>
> > On Nov 12, 2009, at 5:02 PM, Chris Goffinet wrote:
>
> > So I dug more into this (source code). Looks like a documentation bug. I
> > will go fix this.
>

> > *new_thread_per_category*: boolean
>
> >    - If true, will create a new thread for every category seen. Otherwise,

> >    will only create a single thread for every store defined in the
> >    configuration.

> >    - default true

>
> > The input takes, yes or no not boolean.
>
> > On Thu, Nov 12, 2009 at 1:49 PM, Chris Goffinet <c...@chrisgoffinet.com>wrote:
>
> >> Hi
>
> >> We've been experimenting with scribe and noticed an interesting situation.
> >> Lets say for example you have this:
>
> >> <store>
> >> category=Event*
> >> type=network
> >> remote_host=127.0.0.1
> >> remote_port=1463
> >> </store>
>
> >> If I were to send messages with categories such as:
>
> >> Event-1
> >> Event-2

> >> etc, it will create a thread per category seen. I was hoping that *new_thread_per_category=false,
> >> would prevent this, but it does not.*

[TomTom]

Thanks for sending the link Chris. I owe you one.
I vote for evrything being a true/false yes/no.
Thanks a lot guys.
-Tom