Is "max_queue_length" really gone?

[abhinav mithal]

I read though a few posts talking about "max_queue_length" having been
removed from scribe config param list. I still haven't found out what
has replaced it, since "max_queue_length" seems like something scribe
users will want to configure based on the condition of network they
reside on. Here are a few things I am trying hard to find answers to:

1) If "max_queue_length no longer appears in the scribe source", has
it been replaced by something else?
Without "max_queue_length" how does scribe determine when to switch to
a secondary store?

2) I can't find much info about "max_queue_size" except for its
default value (5,000,000 bytes) and that it should be "used for
throttling bussiness"
(https://github.com/facebook/scribe/wiki/Scribe-Configuration)
Is there a relation between "max_queue_size" and good old
"max_queue_length" ?

3) Slightly orthogonal, but during slow network conditions is it
"max_queue_size" which will dictate switching over to secondary
storage?

Many thanks in advance to whoever is nice enough to find time to help.
Abhinav

[Chris Erway]

1) It is true, max_queue_length no longer appears in the scribe source.  Though I did post the commit that you need to revert in another thread last week: https://github.com/facebook/scribe/commit/1b5d5c89a40c737ed7fa9a028f490bf336cd0da8

If you take a look, you'll see that when new messages come in that will make a queue's size exceed max_queue_length, then it sets the BufferStore state to DISCONNECTED, which means messages are sent to the secondary store instead of to the primary store.  Just as the wiki says: if the number of messages in the queue exceeds this value, the buffer store will switch to writing to the secondary store

The secondary store is used whenever the BufferStore state is DISCONNECTED.  If you're curious to find all the ways that can happen, you probably should scan through the BufferStore implementation for "changeState(DISCONNECTED)".  One way the BufferStore can switch to DISCONNECTED mode is if the primary store is unreachable.

2) max_queue_size applies to all stores, not just BufferStores.  max_queue_length only applied to BufferStores.  max_queue_size specifies the maximum number of unhandled messages Scribe will allow for ANY store.  If an incoming message causes a store's queue to exceed max_queue_size, it is thrown away.  The only way you will know when a message has dropped due to max_queue_size is if you check your counters with "scribe_ctrl counters": you will see a counter value appear for the category & the string "denied for queue size".

https://github.com/facebook/scribe/blob/master/src/scribe_server.cpp#L306

3) No, max_queue_size will not cause the secondary store to be used during slow network conditions: in fact, it might even drop your messages under slow network conditions.

I hope someone else will correct me if I'm wrong, but I don't think that there's anything built into BufferStore that will switch to the secondary store under slow network conditions.  I think Scribe may just build up a big queue in front of the slow BufferStore (since the slow primary store is still up, not disconnected) and then start dropping messages due to max_queue_size if the network is too slow to handle the queue.

c

[Chris Erway]

I made the picture seem too bleak in my response: messages that exceed max_queue_size are not simply dropped, scribe returns TRY_LATER for the thrift call to Log.

  https://github.com/facebook/scribe/blob/master/src/scribe_server.cpp#L405

so if you have another scribe running in front of it you can requeue (with must_succeed) and/or put a BufferStore in front of the TRY_LATER.

c

[Inder Pall]

another thing to note is max_queue_size is applicable across all queues for different catgeories.

- inder

--
Thanks,
- Inder
  Tech Platforms @Inmobi
  Linkedin - http://goo.gl/eR4Ub

[abhinav mithal]

Thanks Chris! this is indeed very helpful.
Now I think again, and max_queue_length is perhaps more important than
I thought.
Please correct me if I am wrong:
In case the input queue in scribe server is building up due to primary
store is not reachable/ disconnected. If we have a "max_queue_length"
defined and set to a value smaller than "max_queue_size", it will
start writing to the secondary store ( for eg: a local file system)
and if not prevent, this will delay the input queue from reaching
"max_queue_size" and hence lesser chances of seeing "TRY_LATER" and/or
incoming messages getting dropped on the floor (assuming there is no
BufferStore or another scribe agent to requeue and try later.)

Appreciate your comments.

-Abhinav

On Dec 15, 11:35 pm, Chris Erway <c...@tracelytics.com> wrote:
> I made the picture seem too bleak in my response: messages that exceed max_queue_size are not simply dropped, scribe returns TRY_LATER for the thrift call to Log.

>  https://github.com/facebook/scribe/blob/master/src/scribe_server.cpp#...

>
> so if you have another scribe running in front of it you can requeue (with must_succeed) and/or put a BufferStore in front of the TRY_LATER.
>
> c
>
> On Dec 15, 2011, at 7:21 PM, Chris Erway <c...@tracelytics.com> wrote:
>
>
>
>
>
>
>

> > 1) It is true, max_queue_length no longer appears in the scribe source.  Though I did post the commit that you need to revert in another thread last week:https://github.com/facebook/scribe/commit/1b5d5c89a40c737ed7fa9a028f4...

>
> > If you take a look, you'll see that when new messages come in that will make a queue's size exceed max_queue_length, then it sets the BufferStore state to DISCONNECTED, which means messages are sent to the secondary store instead of to the primary store.  Just as the wiki says: if the number of messages in the queue exceeds this value, the buffer store will switch to writing to the secondary store
>
> > The secondary store is used whenever the BufferStore state is DISCONNECTED.  If you're curious to find all the ways that can happen, you probably should scan through the BufferStore implementation for "changeState(DISCONNECTED)".  One way the BufferStore can switch to DISCONNECTED mode is if the primary store is unreachable.
>
> > 2) max_queue_size applies to all stores, not just BufferStores.  max_queue_length only applied to BufferStores.  max_queue_size specifies the maximum number of unhandled messages Scribe will allow for ANY store.  If an incoming message causes a store's queue to exceed max_queue_size, it is thrown away.  The only way you will know when a message has dropped due to max_queue_size is if you check your counters with "scribe_ctrl counters": you will see a counter value appear for the category & the string "denied for queue size".

> >https://github.com/facebook/scribe/blob/master/src/scribe_server.cpp#...

>
> > 3) No, max_queue_size will not cause the secondary store to be used during slow network conditions: in fact, it might even drop your messages under slow network conditions.
>
> > I hope someone else will correct me if I'm wrong, but I don't think that there's anything built into BufferStore that will switch to the secondary store under slow network conditions.  I think Scribe may just build up a big queue in front of the slow BufferStore (since the slow primary store is still up, not disconnected) and then start dropping messages due to max_queue_size if the network is too slow to handle the queue.
>
> > c
>

[abhinav mithal]

And ofcourse my statement only applies to BufferStore, where
"max_queue_length" will help mitigate such a situation.

[Chris Erway]

Abhinav, I'm not sure if you understood my first response -- unless you have reverted that commit I mentioned, setting max_queue_length will have no effect.  The string "max_queue_length" does not appear anywhere in the scribe source.

[abhinav mithal]

Oh! ofcourse yes, my comments above were true assuming one has
reverted the check-in you have pointed out in your earlier response.

Thanks!

On Dec 16, 12:15 pm, Chris Erway <c...@tracelytics.com> wrote:
> Abhinav, I'm not sure if you understood my first response -- unless you
> have reverted that commit I mentioned, setting max_queue_length will have
> no effect.  The string "max_queue_length" does not appear anywhere in the
> scribe source.
>