FieldGrouping on Multiple Bolts

[anahap]

Hi,


If I have multiple Bolts A and multiple Bolts B (with different
classes A.class and B.class) and I create the following grouping for
Bolt C:

C.fieldsGrouping("A", new Fields("my_field_name")).fieldsGrouping("B",
new Fields("my_field_name"))


will ALL tuples from A AND B that contain the field
my_field_name="zzz" go to the same instance of bolt C, or will they be
grouped by the source bolt?


TIA

[Ben Hughes]

Yes. The task that a tuple goes to is determined by the hashcode of
the fields in the grouping modulo the number of tasks in that bolt.
As long as all fields in the fields grouping are the same and they
return sensible, value based hashcodes, everything will work fine.

The code that handles the grouping is like 5 lines and kind of neat:
https://github.com/nathanmarz/storm/blob/master/src/clj/backtype/storm/daemon/task.clj#L9-13

Ben

[Nathan Marz]

Just to make sure what Ben is saying is clear, in a fields grouping a grouping value will always go to the same consumer task, regardless of the source. Otherwise things like joins of different streams would not be possible. 

--
Twitter: @nathanmarz
http://nathanmarz.com

[ivolo]

In the exact example given above, can the two fields grouping have different stream Ids (or do they have to have the same one) if you want the same "my_field_name" to go to the same task regardless whether its sent from Bolt A or Bolt B?

[Nathan Marz]

Yes they can have different stream ids. The fields grouping equation is hash(subset of tuple fields) % number of consumer tasks. So all that matters is the values of the fields being grouped on.

[invader zim]

Check the docs :

http://nathanmarz.github.com/storm/doc-0.8.1/backtype/storm/topology/InputDeclarer.html#fieldsGrouping(java.lang.String, java.lang.String, backtype.storm.tuple.Fields)

 

Bookmark this: http://nathanmarz.github.com/storm/doc-0.8.1/index.html

 

Cheers,

Zim

On Thursday, 14 March 2013 07:28:57 UTC, Ryan Santos wrote:

Hi, 

I am very new here, I am just wondering how can I assign Stream ids to fieldGrouping?