Disco 0.5 classic worker number of reducers

[Alex Cepoi]

I just upgraded to 0.5.2 and running a cluster with 4 nodes, 12 workers each.

I am using classic workers and I see that most reducers now run only 4 tasks. After a bit of digging I noticed classic worker is implemented as a compatibility over pipelines with a map_shuffle/reduce_shuffle of group_node (this is the only grouping supported for classic workers afais). Thus a map_shuffle phase will only generate 4 output files and the subsequent reduce will only do a max of 4 tasks, which is not good enough for me (a job chained on the previous will also run with only 4 map tasks).

I understood I can manually assign labels to output files to increase this, but I cannot find any sort of example on how this could be done. Is it possible or must I absolutely switch to using pipelines? Any ideas?

[Shayan Pooya]

Hello Alex,

You can use 'partitions' to control the number of reduces (http://disco.readthedocs.org/en/latest/howto/dataflow.html).

Take a look at this blog post: http://continuum.io/blog/literary-chains

This should still work fine in the newer versions of Disco.

I agree that the documentation is not very good in this area.

--
You received this message because you are subscribed to the Google Groups "Disco-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to disco-dev+...@googlegroups.com.
To post to this group, send email to disc...@googlegroups.com.
Visit this group at http://groups.google.com/group/disco-dev.
For more options, visit https://groups.google.com/d/optout.

[Alex Cepoi]

Hey Shayan,

Thanks for the response. I am using partitions=96, but to no avail, the number of reduces is still 4. The same flow worked fine before 0.5.

In the release notes for 0.5 I see: "the number of reduce tasks is determined dynamically, using the user-generated labels attached to task outputs. Hence, it is possible for a job with partitions set to N to have less than N reduce tasks (if, for e.g. there were fewer than N task output labels generated by the maps)".

So, I think due to the fact that map_shuffle and reduce_shuffle output (max) 4 files, I will not get more than 4 subsequent map/reduce tasks. I think I would need support for the other grouping modes in classic worker, or am I getting this wrong?

I am looking here: https://github.com/discoproject/disco/blob/develop/lib/disco/worker/classic/worker.py#L365

[Shayan Pooya]

Hello Alex,

The partitions should still work for the classic worker and if it does not work, it is a bug and we have to fix it. (The number of tasks is determined dynamically, but the classic worker still respects the 'partitions' as much as possible).

I just checked the test_oob and it seems to be working. Take a look at this test. You can run it with:

$ sudo make install-tests

$ disco test test_oob.OOBTestCase.test_chain

In this test, the number of partitions is 10, everything maps to only 2 tasks (maybe because of the hash function?). If you make the number of partitions to 8, then it should create 8 reduces instead of 2.

The place we get the label of the output from the number of partitions is in the default_partition function in worker/classic/func.py

This function should also be improved to work with python 3.

[Terry Peng]

Hi Shayan,

I think the problem are in the later flows. i.e.. in the OOBJob1 we can have 8 reducer by setting the partitions to 8, but if i understand correctly, after disco 0.5 there is group_label in reduce_shuffle after reduce, so OOBJob1 can only have less output file (maximum is the number of disco nodes?), thus the jobs in later stage can only have less mappers, so the whole flow will get slower. 

am i get correct?

[John]

Hello --

I think I'm running into the same problem. I have a job with 128 partitions that gets fed into another job, and the map phase of that second job only ever gets exactly one mapper. It looks as if the reduce-shuffle phase operates on one item -- is there a way around this? I have an upgrade to 0.5.3 from 0.5.2 I still need to do so I apologize if this has been fixed, but I didn't see anything in the issues list.

[Shayan Pooya]

It seems like the reduce_shuffle stage is only added to limit the number of outputs that are given back to the client. Which means, for your usecase it is only hurting the performance. As far as I can say, it is completely safe to disable it:
https://github.com/pooya/disco/commit/57188ba9a962bd3b4e2368ad030a307c872f5804

Please note that by setting the number of partitions, the sort option is not going to work correctly. Currently, there is no warnings generated for this but the output fails to sort correctly. However, it is independent of whether this stage exists or not.

This commits will need further testing, investigation and cleanup to be merged into the main repo.

regards

[jo...@timeburger.com]

Great, thanks very much, and thanks for your work on Disco in general as well!

[Terry Peng]

Great news! eager to see it get merged to master soon!