mvn test fails on virutal box deployment

[Rashmi Vinayak]

Hi All,

I ran "mvn test" on a virtual box deployment just to make sure things are working fine as on local machine, but it fails with an authentication error (message appended below). Are there any additional settings to be configured for deploying on virtual box?

customAuthenticationExactNamePasswordMatchTest(tachyon.security.authentication.AuthenticationUtilsTest)  Time elapsed: 0.17 sec  <<< ERROR!
org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:579)
    at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
    at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
    at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
    at tachyon.security.authentication.AuthenticationUtilsTest.customAuthenticationExactNamePasswordMatchTest(AuthenticationUtilsTest.java:198)

Thanks,
Rashmi

[Bin Fan]

hi Rashmi

Which version of Tachyon are you running? master branch or 0.8 branch?

- Bin

--
You received this message because you are subscribed to the Google Groups "Tachyon Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tachyon-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rashmi K V]

0.8 branch (specifically 0.8.2)

[Yupeng Fu]

Hi Rashmi,

That might be related to a hostname problem on Linux. Can you take a look at /etc/hosts, and see if there's a line:

127.0.0.1 hostname localhost localhost.localdomain

This is problematic because the machine will use the loopback address 127.0.0.1 as its own iP address, which will be confusing to other machines.

You can remove the hostname from it, and let DNS figure it out.

Hope it helps,

--

--Yupeng

[Rashmi K V]

Hi Yupeng,

Yes, there is this following line in the TachyonMaster virtual box:
"127.0.0.1   TachyonMaster localhost localhost.localdomain localhost4 localhost4.localdomain4"

Just to make sure I understand your suggestion, are you suggesting to replace this line with the below line?
"127.0.0.1  localhost localhost.localdomain localhost4 localhost4.localdomain4"

And that I do the same in all the TachyonWorkers as well?

Thanks,

Rashmi

[Yupeng Fu]

Yes, give it a try.

If you just want to run 'mvn test', then you don't need to change anything on TachyonWorkers.

Cheers,

--

--Yupeng

[Rashmi K V]

Hi Yupeng,

Thanks for your suggestion! It helped to get the "mvn test" working on virtual box deployment. I get two tests failing. These tests pass on local machine so they must be related to virtual box deployment. Could you please help how to resolve these?

Test1:
Running tachyon.client.UfsUtilsIntegrationTest
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.687 sec <<< FAILURE!
loadUnderFsTest(tachyon.client.UfsUtilsIntegrationTest)  Time elapsed: 0.687 sec  <<< ERROR!
java.io.IOException: Path /tachyon does not exist
    at tachyon.client.TachyonFSTestUtils.listFiles(TachyonFSTestUtils.java:158)
    at tachyon.client.UfsUtilsIntegrationTest.loadUnderFsTest(UfsUtilsIntegrationTest.java:88)

Test 2:
Running tachyon.network.protocol.RPCMessageIntegrationTest
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.026 sec <<< FAILURE!
tachyon.network.protocol.RPCMessageIntegrationTest  Time elapsed: 0.026 sec  <<< ERROR!
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:463)
    at sun.nio.ch.Net.bind(Net.java:455)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1089)
    at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:430)
    at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:415)
    at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:903)
    at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198)
    at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
    at java.lang.Thread.run(Thread.java:745)

Thanks,

Rashmi

[Gene Pang]

Hi Rashmi,

If you can run the test individually, you will be able to see a clean test log file, and maybe there are more clues in those log files.

in integration-tests/ run:

mvn clean test -Dtest=tachyon.client.UfsUtilsIntegrationTest

and it will produce a log file at integration-tests/target/logs/tests.log

Thanks,

Gene

[Jiří Šimša]

Hi Rashmi,

To follow up on this. Gene's command should be executed from ${TACHYON_HOME}/integration-tests after you successfully build Tachyon. Running the command from ${TACHYON_HOME} will produce an error and running it before Tachyon is successfully built will also produce an error.

Alternatively, you can run the following command from the ${TACHYON_HOME}: "mvn clean test -Dtest=tachyon.client.UfsUtilsIntegrationTest -DfailIfNoTests=false". This command takes a little longer to run because it does not require Tachyon is already built.

Best,

--

Jiří Šimša

--

[Rashmi K V]

Thanks, Gene and Jiri. I ran the tachyon.client.UfsUtilsIntegrationTest test as per Jiri's command line and the following two errors occur first and seem to be leading to the subsequent errors.

2015-12-03 21:02:54,433 ERROR  (MetricsConfig.java:loadConfigFile) - Error loading metrics configuration file.

2015-12-03 21:02:57,727 ERROR  (WorkerFileSystemMasterClient.java:getPinList) -
org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
        at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
        at org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
        at tachyon.thrift.FileSystemMasterService$Client.recv_workerGetPinIdList(FileSystemMasterService.java:680)
        at tachyon.thrift.FileSystemMasterService$Client.workerGetPinIdList(FileSystemMasterService.java:668)
        at tachyon.client.WorkerFileSystemMasterClient.getPinList(WorkerFileSystemMasterClient.java:119)
        at tachyon.worker.block.PinListSync.run(PinListSync.java:86)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

--
You received this message because you are subscribed to a topic in the Google Groups "Tachyon Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tachyon-users/KYmbhX8Th7Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tachyon-user...@googlegroups.com.

[Rashmi K V]

I also ran the other test tachyon.network.protocol.RPCMessageIntegrationTest and get the following in stdout and the log file in this case is empty:

Running tachyon.network.protocol.RPCMessageIntegrationTest
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.984 sec <<< FAILURE!
tachyon.network.protocol.RPCMessageIntegrationTest  Time elapsed: 0.984 sec  <<< ERROR!
java.lang.ExceptionInInitializerError: null
    at tachyon.util.network.NetworkAddressUtils.getLocalIpAddress(NetworkAddressUtils.java:356)
    at tachyon.util.network.NetworkAddressUtils.getLocalHostName(NetworkAddressUtils.java:320)
    at tachyon.conf.TachyonConf.<init>(TachyonConf.java:122)
    at tachyon.conf.TachyonConf.<init>(TachyonConf.java:111)
    at tachyon.Version.<clinit>(Version.java:27)
    at tachyon.Constants.<clinit>(Constants.java:331)
    at tachyon.util.network.NetworkAddressUtils.<clinit>(NetworkAddressUtils.java:53)
    at tachyon.network.protocol.RPCMessageIntegrationTest.beforeClass(RPCMessageIntegrationTest.java:107)

[Yupeng Fu]

Hi Rashmi, 

The error message seems different from your last run:

Running tachyon.network.protocol.RPCMessageIntegrationTest

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.026 sec <<< FAILURE!
tachyon.network.protocol.RPCMessageIntegrationTest  Time elapsed: 0.026 sec  <<< ERROR!
java.net.BindException: Address already in use

I think the reason is you have Tachyon running, and therefore using the port. Can you stop Tachyon (via bin/tachyon-stop.sh) and run the test again?

--

--Yupeng

[Rashmi K V]

Hi Yupeng,

I think the reason is you have Tachyon running, and therefore using the port. Can you stop Tachyon (via bin/tachyon-stop.sh) and run the test again?

Isn't Tachyon expected to be running when I start the virtual box deployment? To make sure I understand your suggestion, do you mean stopping Tachyon on virtual box deployment via bin/tachyon-stop.sh on the virtual box TachyonMaster? Or are you suggesting that Tachyon is running locally and to try stopping it on the local machine (i.e. not on virtual box)?

Thanks,

Rashmi

[Yupeng Fu]

Hi Rashmi,

I guess you're following the guide on tachyon-project website, right? If so, then yes, Tachyon will be deployed and started on the virtual box. 

Usually virtual box is a deployment environment, but not for development. If you just need a sanity check to see if the deployed Tachyon works, then '/bin/tachyon runTests' is sufficient. 'mvn test' is used for running all the tests as part of the development. So, is there any reason that you want to develop in a virtual box? 

Regarding your question, if you really want to run mvn test in virtual box, then you need to stop Tachyon deployment in virtual box (not in the host) first.

Hope it helps.

Cheers,

--

--Yupeng

[Rashmi K V]

Hi Yupeng,

Thanks for the clarifications. I am using virtual box to test a feature that I am working on in a multiple worker scenario. Basically, I want to be able to run BasicOperations like example in a multi-worker scenario. I am not able to do this locally since the guide about running Tachyon locally allows only 1 worker. Is there a way to able to run Tachyon with multiple workers locally (just for sanity check)?

Thanks,

Rashmi

[Yupeng Fu]

Hi Rashmi,

Thanks for providing more info. It makes sense to use virtual box to simulate multiple-worker setup. Could you share more about the feature you're developing, so that we can suggest the best development approach for you? For instance, it doesn't seem to me that you have to do the sanity check in virtual box: you can develop and run the sanity in the host, and then deploy the instances on the guests. Hope it helps.

Cheers,

--

--Yupeng

[Rashmi K V]

Hi Yupeng,

I am working on a research project for which I need to place blocks of a file on different workers when the file is written. This placement is what I am working on now and it is here that I need to test with multiple workers. I am currently developing on my laptop, testing first on local cluster, and then deploying a virtual box cluster on my machine and running some examples (e.g., some classes like BasicOperations) to confirm the code is doing what it is supposed to do. (Thanks to the clarification in your previous email, I see that I don't need to run mvn test on the virtual box cluster.) Please do let me know if you have any suggestions to improve this set up!

Thanks!

Rashmi

[Yupeng Fu]

Hi Rashmi,

Thanks for sharing the info. Yes, I can see why you need virtual boxes for your location-policy feature. And I think your current set up should be good enough for your purpose. Feel free to let me know if you run into any other issues. 

Cheers,

--Yupeng

--

--Yupeng