Installing Crossdata in Mac OSX Elcapitan

[Sambit Dixit]

Hi,

  I downloaded the cross data deb packages and trying to install in macosx for local development and testing. I have already installed deb, fake root, rpm, debianutils through home-brew. Here is the error I'm getting. Note - This works perfectly fine in my linux setup. But wanted to get it running in my macbook for quick testing and demo etc. 

udo dpkg -i /Users/sambit.dixit/Downloads/stratio-crossdata-1.2.1.all.deb 

dpkg: warning: '/usr/local/opt/gnu-tar/bin/gtar' not found in PATH or not executable

Any help would be highly appreciated. 

Regards

-Sambit

[Miguel Angel Fernandez]

Hi Sambit,

the package gnu-tar also has to be installed: 

brew install gnu-tar --with-default-names

We'll update the documentation asap. Thanks for the feedback.

Regards

-Miguel

[Sambit Dixit]

Hi Miguel,
I have done that. Sorry i didnt mention earlier.

Im getting the error even i have installed gnu-tar. Do i need to make any changes anywhere else ?

Regards
-Sambit

[Miguel Angel Fernandez]

Could you, please, check if the path '/usr/local/opt/gnu-tar/bin/gtar' exists? If not, the installation of gtar isn't working properly and it doesn't copy the gtar file to that path correctly 

[Sambit Dixit]

Yes it exists. 

MacBook-Pro:~ sambit.dixit$ /usr/local/opt/gnu-tar/bin/gtar 

gtar: Must specify one of -c, -r, -t, -u, -x

MacBook-Pro:~ sambit.dixit$ 

[migue...@gmail.com]

Can you show us the permissions of the file, please?

[sambit...@olacabs.com]

This was the initial permission when i downloaded. 

-rw-r--r--@  1 sambit.dixit  staff       263328898 Apr  8 14:11 stratio-crossdata-1.2.1.all.deb

I tried next changing the permission for this file. Then tried to run the dpkg as below. 

-rwxrwxrwx@  1 sambit.dixit  staff       263328898 Apr  8 14:11 stratio-crossdata-1.2.1.all.deb

MacBook-Pro:Downloads sambit.dixit$ sudo dpkg -i stratio-crossdata-1.2.1.all.deb 

Password:

Sorry, try again.

Password:

dpkg: warning: '/usr/local/opt/gnu-tar/bin/gtar' not found in PATH or not executable

dpkg: error: 1 expected program not found in PATH or not executable

Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin

MacBook-Pro:Downloads sambit.dixit$ 

[sambit...@olacabs.com]

Even i tried to create a new package from the source code as below. Im getting a fake root error. Have you guys tried this setup in MacOSX already ? Not sure why its coming in my setup. 

For the time being, I have written a shell script to keep all the config and lib jars in one place and then start the main class. This could have been a very simple distribution though to provide a tar ball and then let people run it anywhere any os. 

Here is the steps i followed when i tried from source. 

I have cloned the git repo and followed instructions as in Stratio documentation to create packages. 
I have installed following pre-requisites using home-brew before running 1 and 2 commands below 
a)fakeroot
b)dpkg
c)rpm
d)debianutils
e)gnu-tar 

1. mvn clean install -DskipUTs -DskipITs
2. mvn package -Ppackage -DskipUTs -DskipITs

Im getting below error while running the second command above i.e. mvn package -Ppackage -DskipUTs -DskipITs

INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Stratio Crossdata Parent ........................... SUCCESS [ 0.697 s]
[INFO] Core ............................................... SUCCESS [ 18.055 s]
[INFO] Cassandra Connector ................................ SUCCESS [ 5.932 s]
[INFO] MongoDB Connector .................................. SUCCESS [ 4.087 s]
[INFO] Elasticsearch Connector ............................ SUCCESS [ 8.613 s]
[INFO] Common ............................................. SUCCESS [ 3.165 s]
[INFO] Server ............................................. SUCCESS [ 17.380 s]
[INFO] Driver ............................................. SUCCESS [ 16.199 s]
[INFO] Streaming .......................................... SUCCESS [ 17.463 s]
[INFO] TestsIT ............................................ SUCCESS [ 0.066 s]
[INFO] Dist ............................................... FAILURE [ 2.083 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:34 min
[INFO] Finished at: 2016-06-09T16:18:22+05:30
[INFO] Final Memory: 137M/1431M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.stratio.mojo.unix:unix-maven-plugin:1.2.0:package-deb (deb) on project dist_2.10: Unable to create package. Command 'fakeroot' returned a non-null exit code: 2 -> [Help 1]

Regards

-Sambit

[sambit...@olacabs.com]

Once i'm running the setup (using shell script approach). I'm able to login to cross data shell and followed the instructions to try out samples using Cassandra as in the documentation here - https://stratio.atlassian.net/wiki/display/CROSSDATA1x3/Get+Started. 

 I'm getting an error as below. 

CROSSDATA> show tables

Result for query ID: ed177d54-f108-4aed-af22-541d427dac30

SUCCESS

+---------+-----------+

|tableName|isTemporary|

+---------+-----------+

+---------+-----------+

CROSSDATA> IMPORT TABLES USING com.stratio.crossdata.connector.cassandra OPTIONS (cluster "Test Cluster", spark_cassandra_connection_host '127.0.0.1')

Result for query ID: ac2a2239-e347-4b5a-857b-4602a8f396ea

SUCCESS

+------------------------+-------+

|tableIdentifier         |ignored|

+------------------------+-------+

|[test, shopping_centers]|false  |

|[test, sales]           |false  |

+------------------------+-------+

CROSSDATA> show tables

Result for query ID: 60bf986c-407b-440d-bc01-4409b5f967ba

SUCCESS

+---------------------+-----------+

|tableName            |isTemporary|

+---------------------+-----------+

|test.sales           |false      |

|test.shopping_centers|false      |

+---------------------+-----------+

CROSSDATA> SELECT * FROM test.sales WHERE center = 'Madrid'

Result for query ID: 08e3ea76-aed1-4642-be32-71331063c041

SUCCESS

+------+--------+-----+

|center|products|total|

+------+--------+-----+

|Madrid|51      |7700 |

+------+--------+-----+

CROSSDATA> SELECT * FROM test.sales INNER JOIN test.shopping_centers ON sales.center = shopping_centers.name ORDER BY total

Result for query ID: a3356da1-ab82-4872-ad67-2dc0f1af7289

ERROR

org.apache.spark.sql.AnalysisException: cannot resolve 'sales.center' given input columns: [address, total, opening_year, employees, products, name, center];

CROSSDATA> 

 

Regards

-Sambit

[Miguel Angel Fernandez]

Hi Sambit,

Could you show the permissions of the file /usr/local/opt/gnu-tar/bin/gtar, please?

Could you please run "mvn -X package -Ppackage -DskipUTs -DskipITs" in order to have a verbose output of the maven execution?

I'm El Capitan user and all this command works properly in my Mac.

Nice idea about having a tar ball with a script that can be executed in any os. We'll work on the idea in future releases.

As for the error in the join query execution, Spark 1.6.X doesn't support those qualified column names. Therefore, the query should be:

"SELECT * FROM test.sales INNER JOIN test.shopping_centers ON center = name ORDER BY total"

We'll update the documentation with this change.

Thanks for the feedback

[Sambit Dixit]

Hi Miguel,

   Ah..i didn't realize the Spark 1.6.x problem, thats a good catch.  

Here is the permission as you asked.  

lrwxr-xr-x   1 sambit.dixit  admin      12 Jun  6 14:28 gtar -> /usr/bin/tar

-r-xr-xr-x   1 sambit.dixit  admin  435744 Jun  6 14:20 tar

[Sambit Dixit]

And here is the stack trace after running mvn -X option. 

[ERROR] Failed to execute goal com.stratio.mojo.unix:unix-maven-plugin:1.2.0:package-deb (deb) on project dist_2.10: Unable to create package. Command 'fakeroot' returned a non-null exit code: 2 -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.stratio.mojo.unix:unix-maven-plugin:1.2.0:package-deb (deb) on project dist_2.10: Unable to create package.

at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)

at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)

at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)

at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)

at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)

at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)

at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)

at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)

at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)

at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)

at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)

at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)

at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)

at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)

Caused by: org.apache.maven.plugin.MojoExecutionException: Unable to create package.

at com.stratio.mojo.unix.maven.MojoHelper$Execution.execute(MojoHelper.java:326)

at com.stratio.mojo.unix.maven.plugin.AbstractPackageMojo.execute(AbstractPackageMojo.java:101)

at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)

at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)

... 20 more

Caused by: java.io.IOException: Command 'fakeroot' returned a non-null exit code: 2

at com.stratio.mojo.unix.util.SystemCommand$ExecutionResult.assertSuccess(SystemCommand.java:135)

at com.stratio.mojo.unix.deb.DpkgDeb.execute(DpkgDeb.java:105)

at com.stratio.mojo.unix.maven.deb.DebUnixPackage$DebPreparedPackage.packageToFile(DebUnixPackage.java:213)

at com.stratio.mojo.unix.maven.MojoHelper$Execution.execute(MojoHelper.java:311)

... 23 more

[ERROR] 

[Miguel Angel Fernandez]

Unfortunately, these traces don't show valuable information.

Could you, please, execute also the next command?

mvn package -Ppackage -X -Dmaven.unix.debug=true

It should provide more detailed information about the error

[Sambit Dixit]

Hi Miguel, I have attached the entire console output as debug.txt after running the command with debug options you mentioned.  

[Miguel Angel Fernandez]

Hi Sambit,

I also have a Mac with El Capitan and I'm not able to reproduce the issue. What version of fakeroot do you have installed?

> fakeroot -v

fakeroot version 1.20.2

[Miguel Angel Fernandez]

Hi Sambit,

could you upgrade to the version 1.20.2 instead of the version 1.10.2, please?