write data to nfs/hdfs through alluxio

[Avani Futane]

Hi,

I am using alluxio version 1.8,

I am able to fetch data from hdfs and nfs to alluxio, but while putting data back to nfs/hdfs through alluxio, am not getting proper resources on alluxio site.

Can you please help me with the same.

[Zac Blanco]

Hi Avani,

I assume what you mean by "while putting data back to nfs/hdfs through Alluxio"  you mean that you are writing data through Alluxio? I'm also not sure what you mean by "getting proper resources back on alluxio site". Would you clarify that statement?

If you are not seeing the resources showing up in the under store after writing it is likely due to Alluxio's default WriteType. I recommend reading our architecture page here: https://docs.alluxio.io/os/user/stable/en/Architecture-DataFlow.html#data-flow-write Pay specific attention to the scenarios for different write types.

The default WriteType in Alluxio 1.8 is MUST_CACHE (https://docs.alluxio.io/os/user/1.8/en/reference/Properties-List.html#alluxio.user.file.writetype.default). MUST_CACHE means data is written only to memory and not the under store (HDFS or NFS) in your case.

You have a couple options for getting the data through Alluxio down to the under store.

1. Adjust the write type to be CACHE_THROUGH 

- This will write data to the UFS and Alluxio. This ensures reads of the same data later will be fast. Note this is slower than MUST_CACHE

2. Adjust the write type to THROUGH

- The data you write with this write type is destined only for the under store. It does not go through Alluxio. Note this will slower than MUST_CACHE

3. Adjust the write type to ASYNC_THROUGH

- This WriteType writes to memory first, and then asynchronously persist data to the UFS later on. Note that it is not very mature in v1.8 and I would recommend using v2.0 if you intend to use ASYNC_THROUGH.

4. Manually persist the data with ${ALLUXIO_HOME}/bin/alluxio fs persist <path to data>

- You will be able to write the data very fast with MUST_CACHE, but this command will need to be run manually in order to trigger the persist operations.

Hope this helps. If you clarify your original question some more I can provide better guidance.

Cheers,

Zac

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/alluxio-users/5c9c5815-3ce2-4d3c-9403-0c893f1e647e%40googlegroups.com.

[Avani Futane]

Hi Zac,

Yes, I was trying to write to nfs/hdfs through alluxio. 

By "not getting proper resources on alluxio" I meant I did not find any commands or steps for writing files to hdfs from alluxio.

Thank you for the solution. It worked for nfs. But I am still not able to write to hdfs.

[Zac Blanco]

Thanks for the clarification!

If you're able to write to NFS, then you should be able to write to HDFS as well if you've set up an HDFS mount point. 

Instructions for setting up a root mount with HDFS can be found here: https://docs.alluxio.io/os/user/1.8/en/ufs/HDFS.html If you want a single Alluxio cluster that can mount HDFS and NFS, then you should pick one of them to be the root mount, then manually mount the other as a nested mount. See https://docs.alluxio.io/os/user/1.8/en/advanced/Namespace-Management.html#mounting-under-storage-systems for more information on nested mounts.

If you're looking for instructions on how to perform the writes through Alluxio and down to HDFS, then the information about WriteTypes from the last message still applies. You should be able to use any of the docs listed under the "Data Applications" section.

- MapReduce: https://docs.alluxio.io/os/user/1.8/en/compute/Hadoop-MapReduce.html

- Spark: https://docs.alluxio.io/os/user/1.8/en/compute/Spark.html

- HBase: https://docs.alluxio.io/os/user/1.8/en/compute/HBase.html

- Hive: https://docs.alluxio.io/os/user/1.8/en/compute/Hive.html

- Presto: https://docs.alluxio.io/os/user/1.8/en/compute/Presto.html

If you're also just looking to test small amounts of data locally, you can use the "copyFromLocal" command to write data directly to Alluxio. It will then propagate down to your UFS depending on the WriteType. i.e.

${ALLUXIO_HOME}/bin/alluxio fs copyFromLocal <path to local file> <path in alluxio>

Best,

Zac

--

You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/alluxio-users/587184db-d61a-433b-ad91-76179612ba8f%40googlegroups.com.

[Avani Futane]

Yes, I just had to change the alluxio-site.properties and point the uderfs to hdfs. It works perfectly now.

Thank you for your help.

On Thursday, August 8, 2019 at 2:01:19 PM UTC+5:30, Zac Blanco wrote:

Thanks for the clarification!

If you're able to write to NFS, then you should be able to write to HDFS as well if you've set up an HDFS mount point. 

Instructions for setting up a root mount with HDFS can be found here: https://docs.alluxio.io/os/user/1.8/en/ufs/HDFS.html If you want a single Alluxio cluster that can mount HDFS and NFS, then you should pick one of them to be the root mount, then manually mount the other as a nested mount. See https://docs.alluxio.io/os/user/1.8/en/advanced/Namespace-Management.html#mounting-under-storage-systems for more information on nested mounts.

If you're looking for instructions on how to perform the writes through Alluxio and down to HDFS, then the information about WriteTypes from the last message still applies. You should be able to use any of the docs listed under the "Data Applications" section.

- MapReduce: https://docs.alluxio.io/os/user/1.8/en/compute/Hadoop-MapReduce.html

- Spark: https://docs.alluxio.io/os/user/1.8/en/compute/Spark.html

- HBase: https://docs.alluxio.io/os/user/1.8/en/compute/HBase.html

- Hive: https://docs.alluxio.io/os/user/1.8/en/compute/Hive.html

- Presto: https://docs.alluxio.io/os/user/1.8/en/compute/Presto.html

If you're also just looking to test small amounts of data locally, you can use the "copyFromLocal" command to write data directly to Alluxio. It will then propagate down to your UFS depending on the WriteType. i.e.

${ALLUXIO_HOME}/bin/alluxio fs copyFromLocal <path to local file> <path in alluxio>

Best,

Zac

On Thu, Aug 8, 2019 at 1:11 AM Avani Futane <avani...@abzooba.com> wrote:

Hi Zac,

Yes, I was trying to write to nfs/hdfs through alluxio. 

By "not getting proper resources on alluxio" I meant I did not find any commands or steps for writing files to hdfs from alluxio.

Thank you for the solution. It worked for nfs. But I am still not able to write to hdfs.

On Thursday, August 8, 2019 at 11:59:35 AM UTC+5:30, Avani Futane wrote:

Hi,

I am using alluxio version 1.8,

I am able to fetch data from hdfs and nfs to alluxio, but while putting data back to nfs/hdfs through alluxio, am not getting proper resources on alluxio site.

Can you please help me with the same.

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to alluxi...@googlegroups.com.