Alluxio Fuse problems

[Oleg Mürk]

Hello,

I am having trouble getting Alluxio FUSE to work using:
  http://www.alluxio.org/docs/master/en/Running-Alluxio-on-a-Cluster.html
  http://www.alluxio.org/docs/master/en/Mounting-Alluxio-FS-with-FUSE.html

To be clear - I am connecting to Alluxio cluster from a separate machine.
It seems like I can list files from the master, but getting actual file data fails connecting to the worker. 

Any ideas what I could be doing wrong? Here is the log:

cd ./alluxio-1.2.0

mvn install -Pfuse -DskipTests

echo "ALLUXIO_MASTER_HOSTNAME=dpipe.enterprise01.planetos.com" > conf/alluxio-env.sh

./bin/alluxio fs cat "alluxio://dpipe.enteprise01.planetos.com/rel_0_81x04_indexing/dataset/noaa_gfs_pgrb2_global_0.25degree/reftime/2016-09-15T06-00-00/timevar/time/data.ncml"

* works *

sudo su planetos -c "./bin/alluxio-fuse.sh mount /data/files"

* works *

sudo su - planetos -c "ls /data/files/"

* works *

sudo su - planetos -c "wc /data/files/rel_0_81x04_dataset/transform/ns=/noaa_gfs_pgrb2_global_0.25degree/scheme=/ftp/authority=/ftp.ncep.noaa.gov/path=/pub/data/nccf/com/gfs/prod/gfs.2016091506/gfs.t06z.pgrb2.0p25.f003/chunk=/1/0/data"

* hangs *

cat logs/fuse.log

* see attachment *

[Yupeng Fu]

Hi Oleg,

Are you able to connect to the remote workers? Also, could you paste the logs from alluxio/logs?

Thanks,

Yupeng

Alluxio Inc

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Yupeng Fu]

The tests failed, so I suspect there're issues with under fs setting.

Do you run alluxio master and workers on the machine? And what's your ALLUXIO_UNDERFS_ADDRESS configuration in conf/alluxio-env.sh? 

Yupeng

Alluxio Inc

On Tue, Sep 20, 2016 at 10:51 AM, Oleg Mürk <oleg...@gmail.com> wrote:

Hello!

Here is a run from runTests:
-------

oleg@dpipe:~/alluxio-1.2.0$ ./bin/alluxio runTests

2016-09-20 17:50:46,219 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with FileSystemMasterClient master @ dpipe.enterprise01.planetos.com/10.2.1.94:19998

2016-09-20 17:50:46,258 INFO  type (AbstractClient.java:connect) - Client registered with FileSystemMasterClient master @ dpipe.enterprise01.planetos.com/10.2.1.94:19998

runTest Basic CACHE_PROMOTE MUST_CACHE

2016-09-20 17:50:46,443 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterClient master @ dpipe.enterprise01.planetos.com/10.2.1.94:19998

2016-09-20 17:50:46,455 INFO  type (AbstractClient.java:connect) - Client registered with BlockMasterClient master @ dpipe.enterprise01.planetos.com/10.2.1.94:19998

2016-09-20 17:50:46,492 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to local worker @ dpipe.enterprise01.planetos.com/10.2.1.94:29998

2016-09-20 17:50:46,558 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_PROMOTE_MUST_CACHE took 172 ms.

2016-09-20 17:50:46,639 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_PROMOTE_MUST_CACHE took 80 ms.

Passed the test!

runTest BasicNonByteBuffer CACHE_PROMOTE MUST_CACHE

Passed the test!

runTest Basic CACHE_PROMOTE CACHE_THROUGH

2016-09-20 17:50:46,759 ERROR type (CliUtils.java:runExample) - Exception running test: alluxio.examples.BasicOperations@69a10787

java.io.FileNotFoundException: /data/files/default_tests_files/Basic_CACHE_PROMOTE_CACHE_THROUGH.alluxio.0x4857F4040071EBFC.tmp (No such file or directory)

at java.io.FileOutputStream.open0(Native Method)

at java.io.FileOutputStream.open(FileOutputStream.java:270)

at java.io.FileOutputStream.<init>(FileOutputStream.java:213)

at java.io.FileOutputStream.<init>(FileOutputStream.java:101)

at alluxio.underfs.local.LocalUnderFileSystem.create(LocalUnderFileSystem.java:77)

at alluxio.client.file.FileOutStream.<init>(FileOutStream.java:124)

at alluxio.client.file.BaseFileSystem.createFile(BaseFileSystem.java:102)

at alluxio.examples.BasicOperations.writeFile(BasicOperations.java:85)

at alluxio.examples.BasicOperations.call(BasicOperations.java:72)

at alluxio.examples.BasicOperations.call(BasicOperations.java:42)

at alluxio.cli.CliUtils.runExample(CliUtils.java:51)

at alluxio.cli.TestRunner.runTest(TestRunner.java:148)

at alluxio.cli.TestRunner.runTests(TestRunner.java:121)

at alluxio.cli.TestRunner.main(TestRunner.java:98)

Failed the test!

On Tue, Sep 20, 2016 at 8:31 PM, Yupeng Fu <yup...@alluxio.com> wrote:

Hi Oleg,

I found many failures in the worker.log. Have you been able to run bin/alluxio runTests successfully?

Yupeng

Alluxio Inc

On Tue, Sep 20, 2016 at 9:53 AM, Oleg Mürk <oleg...@gmail.com> wrote:

Hello,

On Tue, Sep 20, 2016 at 6:42 PM, Yupeng Fu <yup...@alluxio.com> wrote:

Are you able to connect to the remote workers? Also, could you paste the logs from alluxio/logs?

On Tue, Sep 20, 2016 at 8:21 AM, Oleg Mürk <oleg...@gmail.com> wrote:

./bin/alluxio fs cat "alluxio://dpipe.enteprise01.planetos.com/rel_0_81x04_indexing/dataset/noaa_gfs_pgrb2_global_0.25degree/reftime/2016-09-15T06-00-00/timevar/time/data.ncml"

* works *

As You can see regular "fs cat" works so it must be connecting to the workers.

* hangs *

cat logs/fuse.log

The client log was attached to the original email. Please see attachment for server logs.

Oleg

[Oleg Mürk]

Hello!

On Tue, Sep 20, 2016 at 8:58 PM, Yupeng Fu <yup...@alluxio.com> wrote:

The tests failed, so I suspect there're issues with under fs setting.

Do you run alluxio master and workers on the machine? And what's your ALLUXIO_UNDERFS_ADDRESS configuration in conf/alluxio-env.sh? 

The tests were failing because Alluxio had only read-only access to underfs (I checked now that they pass when write-access is given).

It seems that running Alluxio fuse client with "sudo" was the source of the original problem, so now I can read the files. 

Some questions:

* Can I run Alluxio server without sudo privileges?

* Is it possible to configure Alluxio FS to work with FUSE "allow_other" option? Currently only one user can read the mount.

* I tried running "tar c /mount | cat > /dev/null" on Alluxio mount and tar reported "file changed as we read it" for some folders. Any ideas what might cause it?

* When running "tar c /mount | cat > /dev/null" Network bandwidth monitors shows very high network traffic on fuse client machine in *both* directions. Any ideas what might cause it?

Thanks!

Oleg

[Yupeng Fu]

Hi Oleg,

Glad that you get it to work. And those are good questions. I'll try my best to answer them.

* Can I run Alluxio server without sudo privileges?

Yes, sudo is used for mounting RamFS. However, there're other mounting options you can use from alluxio-start.sh. 

  MOPT (Mount Option) is one of:

  Mount     Mount the configured RamFS. Notice: this will format the existing RamFS.

  SudoMount Mount the configured RamFS using sudo.

            Notice: this will format the existing RamFS.

  NoMount   Do not mount the configured RamFS.

            Notice: Use NoMount (Linux only) to use tmpFS to avoid sudo requirement.

If you have existing mounted RamFS, you can use mount, or NoMount if you're using Linux

* Is it possible to configure Alluxio FS to work with FUSE "allow_other" option? Currently only one user can read the mount.

That's a good request. I created this ticket to track it: https://alluxio.atlassian.net/browse/ALLUXIO-2231 And you're more than welcome to contribute the feature :)

* I tried running "tar c /mount | cat > /dev/null" on Alluxio mount and tar reported "file changed as we read it" for some folders. Any ideas what might cause it?

I guess it's because when you mount it, only metadata is loaded into Alluxio, but as you read it, the file content is loaded.

* When running "tar c /mount | cat > /dev/null" Network bandwidth monitors shows very high network traffic on fuse client machine in *both* directions. Any ideas what might cause it?

Where is the fuse client running? It's unclear to me why there're bi-direction network transfers. Some more information about the network stats will be helpful.

Hope this helps,

Yupeng

Alluxio Inc

[Yupeng Fu]

Hi Oleg,

From your command, it seems only data download is needed on the client machine. It might be helpful to use some tools like nethogs to find out which process is causing the data transfer. 

Cheers,

Yupeng

Alluxio Inc

On Wed, Sep 21, 2016 at 8:28 AM, Oleg Mürk <oleg...@gmail.com> wrote:

Hello,

On Wed, Sep 21, 2016 at 5:56 PM, Yupeng Fu <yup...@alluxio.com> wrote:

* When running "tar c /mount | cat > /dev/null" Network bandwidth monitors shows very high network traffic on fuse client machine in *both* directions. Any ideas what might cause it?

Where is the fuse client running? It's unclear to me why there're bi-direction network transfers. Some more information about the network stats will be helpful.

I have a separate machine with Alluxio master and worker. Separate machine runs Alluxio FUSE client. Running "tar c /mnt | cat > /dev/null" on it generates 100MB/sec throughput in both directions between Alluxio master/worker and Alluxio fuse client. How much traffic is generated for one random access in a FUSE file?

Oleg

[Oleg Mürk]

The bidirectional traffic stops when I stop reading files and from nethogs it looks that all communication goes with worker port 29999.

Oleg

[Yupeng Fu]

Hi Oleg,

It's still unclear to me. Could you try other commands like cat a simple file, or use bin/alluxio to read a file, and monitor the network traffic?

Yupeng

Alluxio Inc

[Oleg Mürk]

Hello,

Fuse client runs on one node and master+worker on another node. When running this command on fuse client

  find /data.alluxio/files/ -name "file" | xargs -i cat {} | cat > /dev/null

it generates high (100MB/sec) traffic in both directions (ie between fuse client and master+worker node):

This command on fuse client generates traffic only in one direction:

  find /data.alluxio/files/ -name "file" | sed -e "s/^.*files\///" | xargs -i ./bin/alluxio fs cat "alluxio://dpipe.enteprise01.planetos.com/{}" > /dev/null

I even tried adding "fuseOpts.add("-oro");" to file

  integration/fuse/src/main/java/alluxio/fuse/AlluxioFuse.java

but high traffic from fuse client to worker node did not stop.

I have successfully tested "fuseOpts.add("-oallow_other");" though.

Thanks,

Oleg

[Yupeng Fu]

Interesting. It seems like not an issue in Fuse client, but how the command interacts with the client API triggers the traffic in the other direction.

Can you try isolating the issue by not piping the commands but running them individually? For example, I assume  find /data.alluxio/files/ -name "file" should only incur one-direction traffic.

Cheers,

Yupeng

Alluxio Inc

[Oleg Mürk]

Hello

On Wed, Sep 28, 2016 at 7:24 PM, Yupeng Fu <yup...@alluxio.com> wrote:

Interesting. It seems like not an issue in Fuse client, but how the command interacts with the client API triggers the traffic in the other direction.

Can you try isolating the issue by not piping the commands but running them individually? For example, I assume  find /data.alluxio/files/ -name "file" should only incur one-direction traffic.

So I ran two separate queries:

  find /data.alluxio/files/ -name "file" > files.log

  cat files.log | xargs -i cat {} | cat > /dev/null
High bidirectional traffic is generated by the latter.

Thanks,

Oleg

[Yupeng Fu]

Hi Oleg,

Glad you found the problem. I think using a temp file locally will avoid the bidirectional traffic. I suspect the issue was because the temp file is under the mount point, so the command performs write to alluxio.

Cheers,

Yupeng

Alluxio Inc

[Oleg Mürk]

Hello,

On Tue, Oct 4, 2016 at 6:06 PM, Yupeng Fu <yup...@alluxio.com> wrote:

Glad you found the problem. I think using a temp file locally will avoid the bidirectional traffic. I suspect the issue was because the temp file is under the mount point, so the command performs write to alluxio.

I actually didn't find the solution. This command generates high bidirectional traffic:

 cat files.log | xargs -i cat {} | cat > /dev/null

files.log is not on alluxio mount. The traffic is generated by cat command run for files on fuse mount.

Oleg

[Yupeng Fu]

Hi Oleg,

Just to confirm, you called cat under the mount point? It seems cat invokes the Alluxio's fuse API, but if files.log is not under alluxio mount then you can cat it outside the mount point and not invoke fuse.

Yupeng

Alluxio Inc

--

[Oleg Mürk]

To my best understanding files.log contains a list of files on alluxio mount point. I pass them one-by-one using xargs to "cat" which reads them and eventually writes to /dev/null.

Oleg