Optimising Cascading Codes.

[Shivank Garg]

Hello everyone, It is been an year since I have been working in cascading and with some ups and downs everything seems to have worked out. But now I have to revisit my jobs to enhance  their running time from like 1hr, 0.5hr to less than 15 mins. I ask everyone if they have some suggestions on how to optimise our cascading codes, what are the best practices to write efficient codes. Share links to any such documents too. Would love if Chris, you could share some insights on development in cascading jobs.

[Ken Krugler]

Hi Shivank,

As Bill Atkinson told me when I first started working at Apple, "Measure, then optimize"

Which means generating a lot of stats from the job, then staring at the numbers for a while to figure out where your bottlenecks are.

E.g. is it because you have an unbalanced data set, and one reduce task is taking a long time to complete, which then blocks downstream work from starting?

Without the stats you don’t really know…

— Ken

On Jul 4, 2018, at 4:39 AM, Shivank Garg <shivan...@gwynniebee.com> wrote:

Hello everyone, It is been an year since I have been working in cascading and with some ups and downs everything seems to have worked out. But now I have to revisit my jobs to enhance  their running time from like 1hr, 0.5hr to less than 15 mins. I ask everyone if they have some suggestions on how to optimise our cascading codes, what are the best practices to write efficient codes. Share links to any such documents too. Would love if Chris, you could share some insights on development in cascading jobs.

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at https://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/42a64234-e2cc-4b77-9d34-fe859bce94f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--------------------------------------------

http://about.me/kkrugler

+1 530-210-6378