"There is no worker with enough space for a new block of size ..."

[Jim Carroll]

This is probably a dumb question but I keep getting this error message: "There is no worker with enough space for a new block of size 536,870,912"

I'm running Alluxio (1.7.0) clustered with an hdfs backing store alongside of yarn.

I can see files through alluxio if I put them in hdfs but I can't run the tests or put anything into the filesystem without it failing with that message.

E.g. : `./bin/alluxio fs copyFromLocal /tmp/somefile /` gives that error.

Any help would be appreciated.

Thanks.

Jim

[Jim Carroll]

More information 

1) Alluxio 1.7.0

2) Hadoop 2.7

3) Ubuntu 16.04 OS on all hosts

4) Java openjdk version "1.8.0_151"

 

[Andrew Audibert]

Hey Jim,

Can you look at the master UI at masterhost:19999 under the workers tab and share whether your workers are connected to the master, and how much space is free on your workers? If the workers aren't connected, I suggest looking for the root cause in their logs (under ${ALLUXIO_HOME}/logs/worker.log or ${ALLUXIO_HOME}/logs/worker.out by default).

Hope that helps,

Andrew

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jim Carroll]

Thanks Andrew.

I must be missing something in the configuration since the workers, while there's no error in the logs, don't appear to be registered in the master. I have the conf/masters and conf/workers set the same on all hosts and filled with the appropriate IP addresses. When I start the master using:

./bin/alluxio/alluxio-start.sh all NoMount

... it starts the workers up on the other hosts but the list in the UI is empty (and I have 0B capacity on workers).

My alluxio-site.properties is:

# Common properties

alluxio.master.hostname=[master ip]

alluxio.underfs.hdfs.configuration=/usr/local/hadoop/etc/hadoop/core-site.xml:/usr/local/hadoop/etc/hadoop/hdfs-site.xml

alluxio.underfs.address=hdfs://[namenode ip]:9000/

# Security properties

alluxio.security.authorization.permission.enabled=false

# Worker properties

alluxio.worker.memory.size=4GB

... everything else is commented out. Also, if it matters, I can see files in the underlying file system when I use:

./bin/alluxio fs ls /

[Jim Carroll]

Ugh. I think I see the problem. The user that alluxio is running as doesn't have access to the tiered storage. I thought that was disabled with NoMount but I guess I misunderstood.

[and...@alluxio.com]

Ah that explains it - NoMount requires that the ramdisk is already mounted, and that the current process has permissions to read and write to it.