documentation for shardSpec?

[Prashant Deva]

There doesn't seem to be ay documentation at all for shardSpec?

I assume without it, one cannot run more than one realtime node at a time.

Can someone point me to some documentation for shardSpec or provide an explanation in this thread itself?

[Gian Merlino]

Hmm, it looks like you're right. That is definitely an oversight and we should have docs there.

You can imagine all data in a druid cluster being arranged over a timeline. Each segmentGranularity region of your timeline has a set of segments that contains the data for that time region. So if your segmentGranularity is "hour" then each hour might have 1 or 2 or any number of segments that form a complete set. Each of those segments has the same version number, but different partition numbers. The partitionNums start at 0 and go up as far as necessary. When the broker receives a query for a particular hour, it will find the most recent segment version for that hour, then find all partitionNums associated with that highest version, then send the query to nodes that are currently serving those segments, and finally merge the results. If it finds multiple segments with the same partitionNum, it will just pick one of them. But it will query all unique partitionNums it finds. This is because it makes an assumption that two segments for the same datasource, interval, version, and partitionNum actually contain the same data and only exist as separate segments for purposes of redundancy.

In batch indexing, the DeterminePartitions job first determines how many segments need to be created for each segmentGranularity region, and then the IndexGenerator job actually creates those segments and assigns them unique partitionNums. The historical nodes then load the segments with whatever replication factor you have configured.

In realtime indexing, each realtime node (or realtime task, in the indexing service) creates and serves segments with a single partitionNum. The shardSpecs control the manner in which those segments are advertised. There are a few different options, but most people choose linear shardSpecs, where all you need to specify is the partitionNum: {"type":"linear","partitionNum":0}. Because of the way the brokers make queries, you can use shardSpecs both to create redundancy and to scale out ingestion. To create redundancy, you should create two or more realtime nodes/tasks with the same shardSpec and the same underlying data (since the broker will assume same partitionNum = same data, and will only query one of them). To scale out ingestion, you should create two or more realtime nodes/task with different shardSpecs and different underlying data (since the broker will query all of them and merge the results). You can also do both.

[Prashant Deva]

> There are a few different options, but most people choose linear
>

What other options are there?

Prashant

Sent from my iPhone

[Gian Merlino]

The other ones that make most sense for realtime are none and numbered (there are some others useful for batch, but less useful for realtime). "none" is like linear with a fixed partitionNum of 0, so doesn't really confer much benefit, but it does exist. "numbered" is like linear except for one difference: if there is an incomplete set of segments out there (like, partitionNums 0 and 2 are available, but 1 is unavailable) then with linear shards, the broker will query 0 and 2 and skip 1; with numbered shards, the broker will skip *all* of them and act as if there is no data available for the interval.

[Prashant Deva]

Gian,

 Can you give an explanation of all the possible options for shardspec. since there is no official documentation on this maybe this thread can serve as one.

or maybe you can contribute it to druid docs on github

Prashant

--
You received this message because you are subscribed to a topic in the Google Groups "Druid Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-development/GFBM4ifA6ac/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/3039f8d4-6f5b-4df8-920a-4d67e1727d16%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Gian Merlino]

IMO the only ones that make sense to use yourself are linear and numbered. The others are mostly for internal use. I think linear is generally best, unless you want the all-or-nothing results behavior, in which case numbered is best. Linear looks like {"type":"linear", "partitionNum":0}. Numbered looks like {"type":"numbered", "partitionNum":0, "partitions": 2} (it needs to know the total number of partitions in advance).

I filed this github issue to remind us to add official docs: https://github.com/metamx/druid/issues/706