How data blocks are partitioned

[ems eril]

Hi Team ,

    I have simple question about how data is partitioned . If we have 2 nodes each with 80 GB total RAM and we allocate 50GB for each node what happens when you try to store a file that is 60GB or more ? How does partitioning rule work in Alluxio ?

Thanks

-pedram

[Andrew Audibert]

Hi pedram,

Alluxio uses location policies to decide how to partition blocks. By default Alluxio will write to the local worker if there is one, otherwise it will choose a random worker. The copyFromLocal shell command is an exception; by default it will partition the file's blocks evenly across all workers.

-Andrew

--
You received this message because you are subscribed to the Google Groups "Alluxio Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Andrew Audibert

Alluxio, Inc. | Alluxio Open Source | Alluxio Community Site

[Andrew Audibert]

Hi pedram,

Can you share more details about your usage? What behavior are you seeing, and what behavior do you want to get?

You are right, spreading blocks evenly across all workers will lead to network-speed reads instead of memory-speed. This can still give performance improvements, especially if the network is fast or reading from the UFS is slow.

-Andrew

On Fri, Nov 16, 2018 at 8:16 AM ems eril <emsr...@gmail.com> wrote:

Hi Andrew ,

    What do you mean " by default it will partition the file's blocks evenly across all workers. " ? If its partitioning it evenly I would think reads would then need to go across all workers including locale and would defeat the memory speed where aiming for correct ? 

Thanks