Conf Aluxio for Spark and HDFS

Cam Mach

unread,

Dec 1, 2017, 7:47:25 PM12/1/17

to Alluxio Developers

Hello everyone,

I am trying to configure Aluxio to work with Spark and Hadoop. I am following this blog: https://www.alluxio.org/docs/master/en/Running-Spark-on-Alluxio.html, but I have difficulty in find the core-site.xml conf file. I am running Hadoop &Spark 2.7, so guess that file is no longer available. Then, how can I configure Aluxio for Spark and Hadoop 2.7, or where can I find core-site.xml?

Thanks,

Cam

Bin Fan

unread,

Dec 1, 2017, 8:43:22 PM12/1/17

to Cam Mach, Alluxio Developers

hi Cam,

To configure Alluxio with Hadoop and Spark, typically you don't really need to update core-site.xml,

unless you want to use HDFS 1.x. It seems you want to use 2.7. So feel free to skip any instructions

related to core-site.xml described in https://www.alluxio.org/docs/master/en/Running-Spark-on-Alluxio.html

- Bin

--
You received this message because you are subscribed to the Google Groups "Alluxio Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

- Bin Fan

Software Engineer

Alluxio Inc

www.alluxio.com

Cam Mach

unread,

Dec 2, 2017, 1:25:08 PM12/2/17

to Alluxio Developers

Hi Bin,

Thank you for your response. I skipped that step of configuration (core-site.xml), but Alluxio can't see the files in my hdfs. I went back to the document, and saw it keeps repeating asking for that conf one, especially that last section of the doc, so I tried it, but it still doesn't work for me (I found the core-site.xml in my fs). I'm just trying to use hdfs with Alluxio. I am new to Alluxio, so hope it won't bug you guys.

Thank you,

On Friday, December 1, 2017 at 5:43:22 PM UTC-8, Bin Fan wrote:

hi Cam,

To configure Alluxio with Hadoop and Spark, typically you don't really need to update core-site.xml,
unless you want to use HDFS 1.x. It seems you want to use 2.7. So feel free to skip any instructions
related to core-site.xml described in https://www.alluxio.org/docs/master/en/Running-Spark-on-Alluxio.html

- Bin

On Fri, Dec 1, 2017 at 4:47 PM, Cam Mach <camm...@gmail.com> wrote:

Hello everyone,
I am trying to configure Aluxio to work with Spark and Hadoop. I am following this blog: https://www.alluxio.org/docs/master/en/Running-Spark-on-Alluxio.html, but I have difficulty in find the core-site.xml conf file. I am running Hadoop &Spark 2.7, so guess that file is no longer available. Then, how can I configure Aluxio for Spark and Hadoop 2.7, or where can I find core-site.xml?

Thanks,
Cam

--
You received this message because you are subscribed to the Google Groups "Alluxio Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Cam Mach

unread,

Dec 4, 2017, 12:40:49 AM12/4/17

to Bin Fan, Alluxio Developers

Hi Bin,

Thank you so much for your response, again. Please see my comments in line.

Regards,

On Sun, Dec 3, 2017 at 7:07 PM, Bin Fan <bin...@alluxio.com> wrote:

Put my replies inline.

On Sat, Dec 2, 2017 at 10:25 AM, Cam Mach <camm...@gmail.com> wrote:
Hi Bin,

Thank you for your response. I skipped that step of configuration (core-site.xml), but Alluxio can't see the files in my hdfs.

Can you elaborate here what you mean by "Alluxio can't see the files in my hdfs."

==> Cam: as it's documented, if I created a directory called "alluxio" in hdfs, files I put in it, will be visible in alluxio, (when running command bin/alluxio fs ls -R /). I am not sure if it still works in 1.6?

e.g., if you are running Alluxio 1.6, can you run
$ bin/alluxio mount
and see the mount point info
Also, do you run
$ bin/alluxio fs ls /PATH/TO/YOUR_HDFS_MOUNT

==> Cam: I also tried to mount hdfs directory with Alluxio directory, but it throws me an error. Here is the command I ran, and error message it shows:

[cam@camdev02 alluxio-1.6.1]$ sudo bin/alluxio fs mount -readonly alluxio://localhost:19998/mtn/hdfs hdfs://localhost:9000/alluxio

File alluxio://localhost:19998/mtn/hdfs creation failed. Component 1(mtn) does not exist

(I already created .../mnt/hdfs in Alluxio, but don't know why it keeps complaining)

Before running a ls command, Alluxio will not automatically discover the files from HDFS

I went back to the document, and saw it keeps repeating asking for that conf one,

"saw it keeps repeating asking for that conf one,"

I don't understand what you mean here. Can you explain a bit more here.
What keeps asking for the conf?

especially that last section of the doc, so I tried it, but it still doesn't work for me (I found the core-site.xml in my fs). I'm just trying to use hdfs with Alluxio. I am new to Alluxio, so hope it won't bug you guys.

DO you mean the section of "Additional Setup for HDFS"?
this is only required when you are using HDFS 1.x

===> Cam: Yes, I means that section.

Thank you,

On Friday, December 1, 2017 at 5:43:22 PM UTC-8, Bin Fan wrote:
hi Cam,

To configure Alluxio with Hadoop and Spark, typically you don't really need to update core-site.xml,
unless you want to use HDFS 1.x. It seems you want to use 2.7. So feel free to skip any instructions
related to core-site.xml described in https://www.alluxio.org/docs/master/en/Running-Spark-on-Alluxio.html

- Bin

On Fri, Dec 1, 2017 at 4:47 PM, Cam Mach <camm...@gmail.com> wrote:
Hello everyone,
I am trying to configure Aluxio to work with Spark and Hadoop. I am following this blog: https://www.alluxio.org/docs/master/en/Running-Spark-on-Alluxio.html, but I have difficulty in find the core-site.xml conf file. I am running Hadoop &Spark 2.7, so guess that file is no longer available. Then, how can I configure Aluxio for Spark and Hadoop 2.7, or where can I find core-site.xml?

Thanks,
Cam

--
You received this message because you are subscribed to the Google Groups "Alluxio Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
- Bin Fan

Software Engineer
Alluxio Inc
www.alluxio.com

--
You received this message because you are subscribed to the Google Groups "Alluxio Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Cam Mach

unread,

Dec 4, 2017, 12:59:52 PM12/4/17

to Bin Fan, Alluxio Developers

Thanks Bin, It works for me now.

Cam Mach

Software Engineer

E-mail: camm...@gmail.com

Tel: 206 972 2768

On Sun, Dec 3, 2017 at 9:48 PM, Bin Fan <bin...@alluxio.com> wrote:

put my replies inline

On Sun, Dec 3, 2017 at 9:40 PM, Cam Mach <camm...@gmail.com> wrote:
Hi Bin,

Thank you so much for your response, again. Please see my comments in line.

Regards,

On Sun, Dec 3, 2017 at 7:07 PM, Bin Fan <bin...@alluxio.com> wrote:
Put my replies inline.

On Sat, Dec 2, 2017 at 10:25 AM, Cam Mach <camm...@gmail.com> wrote:
Hi Bin,

Thank you for your response. I skipped that step of configuration (core-site.xml), but Alluxio can't see the files in my hdfs.
Can you elaborate here what you mean by "Alluxio can't see the files in my hdfs."
==> Cam: as it's documented, if I created a directory called "alluxio" in hdfs, files I put in it, will be visible in alluxio, (when running command bin/alluxio fs ls -R /). I am not sure if it still works in 1.6?

This should still work

e.g., if you are running Alluxio 1.6, can you run
$ bin/alluxio mount
and see the mount point info
Also, do you run
$ bin/alluxio fs ls /PATH/TO/YOUR_HDFS_MOUNT
==> Cam: I also tried to mount hdfs directory with Alluxio directory, but it throws me an error. Here is the command I ran, and error message it shows:
[cam@camdev02 alluxio-1.6.1]$ sudo bin/alluxio fs mount -readonly alluxio://localhost:19998/mtn/hdfs hdfs://localhost:9000/alluxio
File alluxio://localhost:19998/mtn/hdfs creation failed. Component 1(mtn) does not exist
(I already created .../mnt/hdfs in Alluxio, but don't know why it keeps complaining)

Can you remove /mnt/hdfs from Alluxio first?
Basically with Alluxio mount cmd, it requires the mount destination dir not exist, and the command will create that dir.

Before running a ls command, Alluxio will not automatically discover the files from HDFS

I went back to the document, and saw it keeps repeating asking for that conf one,

"saw it keeps repeating asking for that conf one,"
I don't understand what you mean here. Can you explain a bit more here.
What keeps asking for the conf?

especially that last section of the doc, so I tried it, but it still doesn't work for me (I found the core-site.xml in my fs). I'm just trying to use hdfs with Alluxio. I am new to Alluxio, so hope it won't bug you guys.

DO you mean the section of "Additional Setup for HDFS"?
this is only required when you are using HDFS 1.x
===> Cam: Yes, I means that section.

Are you using HDFS 1.x? if not, just ignore the entire section of "Additional Setup for HDFS"
as it talks about using HDFS 1.x / HDFS in HA

Bin Fan

unread,

Dec 5, 2017, 8:51:33 AM12/5/17

to Cam Mach, Alluxio Developers

Put my replies inline.

On Sat, Dec 2, 2017 at 10:25 AM, Cam Mach <camm...@gmail.com> wrote:

Hi Bin,

Thank you for your response. I skipped that step of configuration (core-site.xml), but Alluxio can't see the files in my hdfs.

Can you elaborate here what you mean by "Alluxio can't see the files in my hdfs."

e.g., if you are running Alluxio 1.6, can you run

$ bin/alluxio mount

and see the mount point info

Also, do you run

$ bin/alluxio fs ls /PATH/TO/YOUR_HDFS_MOUNT

Before running a ls command, Alluxio will not automatically discover the files from HDFS

I went back to the document, and saw it keeps repeating asking for that conf one,

"saw it keeps repeating asking for that conf one,"

I don't understand what you mean here. Can you explain a bit more here.

What keeps asking for the conf?

especially that last section of the doc, so I tried it, but it still doesn't work for me (I found the core-site.xml in my fs). I'm just trying to use hdfs with Alluxio. I am new to Alluxio, so hope it won't bug you guys.

DO you mean the section of "Additional Setup for HDFS"?

this is only required when you are using HDFS 1.x

Thank you,

On Friday, December 1, 2017 at 5:43:22 PM UTC-8, Bin Fan wrote:
hi Cam,

To configure Alluxio with Hadoop and Spark, typically you don't really need to update core-site.xml,
unless you want to use HDFS 1.x. It seems you want to use 2.7. So feel free to skip any instructions
related to core-site.xml described in https://www.alluxio.org/docs/master/en/Running-Spark-on-Alluxio.html

- Bin

On Fri, Dec 1, 2017 at 4:47 PM, Cam Mach <camm...@gmail.com> wrote:
Hello everyone,
I am trying to configure Aluxio to work with Spark and Hadoop. I am following this blog: https://www.alluxio.org/docs/master/en/Running-Spark-on-Alluxio.html, but I have difficulty in find the core-site.xml conf file. I am running Hadoop &Spark 2.7, so guess that file is no longer available. Then, how can I configure Aluxio for Spark and Hadoop 2.7, or where can I find core-site.xml?

Thanks,
Cam

--
You received this message because you are subscribed to the Google Groups "Alluxio Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
- Bin Fan

Software Engineer
Alluxio Inc
www.alluxio.com

--
You received this message because you are subscribed to the Google Groups "Alluxio Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Bin Fan

unread,

Dec 5, 2017, 8:51:33 AM12/5/17

to Cam Mach, Alluxio Developers

put my replies inline

On Sun, Dec 3, 2017 at 9:40 PM, Cam Mach <camm...@gmail.com> wrote:

Hi Bin,

Thank you so much for your response, again. Please see my comments in line.

Regards,

On Sun, Dec 3, 2017 at 7:07 PM, Bin Fan <bin...@alluxio.com> wrote:
Put my replies inline.

On Sat, Dec 2, 2017 at 10:25 AM, Cam Mach <camm...@gmail.com> wrote:
Hi Bin,

Thank you for your response. I skipped that step of configuration (core-site.xml), but Alluxio can't see the files in my hdfs.
Can you elaborate here what you mean by "Alluxio can't see the files in my hdfs."
==> Cam: as it's documented, if I created a directory called "alluxio" in hdfs, files I put in it, will be visible in alluxio, (when running command bin/alluxio fs ls -R /). I am not sure if it still works in 1.6?

This should still work

e.g., if you are running Alluxio 1.6, can you run
$ bin/alluxio mount
and see the mount point info
Also, do you run
$ bin/alluxio fs ls /PATH/TO/YOUR_HDFS_MOUNT
==> Cam: I also tried to mount hdfs directory with Alluxio directory, but it throws me an error. Here is the command I ran, and error message it shows:
[cam@camdev02 alluxio-1.6.1]$ sudo bin/alluxio fs mount -readonly alluxio://localhost:19998/mtn/hdfs hdfs://localhost:9000/alluxio
File alluxio://localhost:19998/mtn/hdfs creation failed. Component 1(mtn) does not exist
(I already created .../mnt/hdfs in Alluxio, but don't know why it keeps complaining)

Can you remove /mnt/hdfs from Alluxio first?

Basically with Alluxio mount cmd, it requires the mount destination dir not exist, and the command will create that dir.

Before running a ls command, Alluxio will not automatically discover the files from HDFS

I went back to the document, and saw it keeps repeating asking for that conf one,

"saw it keeps repeating asking for that conf one,"
I don't understand what you mean here. Can you explain a bit more here.
What keeps asking for the conf?

especially that last section of the doc, so I tried it, but it still doesn't work for me (I found the core-site.xml in my fs). I'm just trying to use hdfs with Alluxio. I am new to Alluxio, so hope it won't bug you guys.

DO you mean the section of "Additional Setup for HDFS"?
this is only required when you are using HDFS 1.x
===> Cam: Yes, I means that section.

Are you using HDFS 1.x? if not, just ignore the entire section of "Additional Setup for HDFS"

as it talks about using HDFS 1.x / HDFS in HA

Reply all

Reply to author

Forward