File not fully loaded in cache

[Pascal Gillet]

Alluxio 1.7.1-hadoop-2.8

Mesos 1.4.1

Ceph Luminous for DFS and Object Storage with S3 interface 

Centos 7.3.1611

JDK 1.8.0_131

Spark 2.2.1

Hi,

I try to run a simple Spark Job in a Mesos cluster to test Alluxio as described in https://www.alluxio.com/blog/getting-started-with-alluxio-and-spark.

Whenever I use ./bin/alluxio fs load /path/to/bigTextFile or an Alluxio URL in my Spark job to fetch a big text file from the S3 backend, the said file is not fully loaded into memory, while there is plenty of room across my workers:

sc = SparkContext()

file = sc.textFile('alluxio://path/to/bigTextFile')

print(file.count())

Why is that?

Do I miss some obvious Alluxio configuration?

Thanks,

Pascal GILLET

[Gene Pang]

Hi Pascal,

I have a few questions about your environment:

- Does this spark job succeed?

- Is any portion of the file in memory?

- Is Alluxio deployed on mesos? which containerizer?

- How many Alluxio workers?

Thanks,

Gene

[Gene Pang]

Hi Pascal,

I think what is happening is that the loading is writing to only a single machine. Since a single worker does not have space for the whole file, it does not fully get loaded into memory.

This can be controlled by the write location policy. The default is LocalFirstPolicy, but you can change it to a different policy. More documentation about the location policy is here: https://www.alluxio.org/docs/1.7/en/Clients-Alluxio-Java.html#location-policy

Thanks,

Gene

On Fri, Apr 27, 2018 at 7:57 AM, Pascal Gillet <pascal...@gmail.com> wrote:

Other details:

- My file is never totally loaded in memory even with ./bin/alluxio fs load /path/to/bigTextFile

- The worker memory size is 2 GB * 6 ~ 8 workers: 12 GB min ~ 16 GB max total 

2018-04-27 16:48 GMT+02:00 Pascal Gillet <pascal...@gmail.com>:

Hi Gene,

- Yes, the Spark job succeeds but it takes as much time as if it reads the file directly from the S3 backend.

- Yes, I have a 8GB file that is never totally loaded in memory: from 25% to 50%, if I run the job multiple times, and I can see that the data in memory sometimes moves from workers to others.

- Alluxio is deployed with High Availability with ZooKeeper but not through Mesos: I have 3 Alluxio masters on separate servers, 3 Mesos masters, and the Alluxio workers are also the Mesos agents (whose resources are partitioned for Mesos and Alluxio)

- My Spark job is packaged in a Docker container executed through Mesos: I mounted the Alluxio client jar in the Spark classpath, and I also had to mount /mnt/ramdisk, otherwise Spark complains about FileNotFoundExceptions 

- 6 ~ 8 Alluxio Workers (Mesos agents)

Thanks,

Pascal

--
You received this message because you are subscribed to a topic in the Google Groups "Alluxio Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/alluxio-users/dQYdzrgBb6M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to alluxio-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Pascal Gillet]

Hi Gene,

Yes, that was it!

The property was under my eyes all the time...

Thank you very much,

Pascal

[Pascal Gillet]

Hi Gene,

I also started the Alluxio HTTP proxy on all my workers to interact with Alluxio using the S3 API with a boto client.

I added the alluxio.user.file.write.location.policy.class=alluxio.client.file.policy.RoundRobinPolicy in the configuration file on all my workers.

As the proxy becomes a client itself, I would have expected that this user specific property would be applicable and that all the files downloaded through the proxy would be completely loaded in memory and scattered across the workers.

But only some files are loaded on the proxy host.

Did I miss another configuration?

Thanks,

Pascal

[Gene Pang]

Hi Pascal,

If you use that configuration parameter on the client itself, does that change the behavior?

Thanks,

Gene

[Gene Pang]

Hi Pascal,

Were you able to resolve your issue?

Thanks,

Gene

[Pascal Gillet]

Hi Gene,

Thank you for coming back to me.

Actually, no, because I don't know how to/can't pass the alluxio.user.file.write.location.policy.class property from my boto3 client (I do not use the Python Alluxio client).

Thanks,

Pascal

--