DiskErrorException: Could not find any valid local directory for dag

[pk.4...@gmail.com]

Hello Sungwoo,

The following queries are failing on  :

SELECT DISTINCT <col-name> FROM <table-or-view-name>;

select * from <table-name> order by <column-name>;

The error:

[Code: 2, SQL State: 08S01] Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Terminating unsuccessfully: Vertex failed, vertex_37982451_0000_5_01, Task unsuccessful: Map 1, task_37982451_0000_5_01_000000, org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for dag_5/container_K@9/vertex_1/attempt_37982451_0000_5_01_000000_0_10003_0/file.out

Log attached. 

Fyi : We are using

1. HiveMR3 on K8S

2. S3 for data storage

3. Postgres as SQL engine

4. NVMe for ephemeral storage for workers

5. All Workers are fine - up and ready -- that means ephemeral storage (hostpath) has been mounted correctly. 

6. Still on MR3-1.3-snapshot (a custom build w/ a few hive related fixes)

Please let me know where to check the issue and how to fix it. 

Thanks. A quick response as usual would be highly appreciated.

[Sungwoo Park]

Hello,

The error message says that a ContainerWorker Pod fails to create a file under the local work directory (specified by mr3.k8s.pod.worker.hostpaths). To find out why the error occurs, I would:

1) manually find the ContainerWorker Pod where the error is generated

2) go to the local work directory, and see if new files can be created (dag_5/container_K@9/vertex_1/attempt_37982451_0000_5_01_000000_0_10003_0/file.out in your case) and if enough disk space is available.

Note that LocalDirAllocator.java checks the size of remaining disk space. Hence, if your query generates huge intermediate files and the local work directory cannot accommodate them, the same error is generated. For example, 'select * from <table-name> order by <column-name>;' might generate huge intermediate files because it dumps the entire table. If a small query is finished successfully, I guess this might be the cause.

Hope this helps,

--- Sungwoo

--
You received this message because you are subscribed to the Google Groups "MR3" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hive-mr3+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hive-mr3/73991648-5727-43ac-b3a4-53466108a665n%40googlegroups.com.

[pk.4...@gmail.com]

Thanks Sungwoo for the prompt response.

I checked the local work dir on worker (mr3.k8s.pod.worker.hostpaths) and found ti has been set to use mounted disk /ephemeral as seen below:

------------------------

<property>
<name>mr3.k8s.pod.worker.hostpaths</name>
<value>/ephemeral</value>
</property>

<property>
<name>mr3.k8s.master.local.dir.persistentvolumes</name>
<value></value>
</property>

---------------------------

The  content of /ephemeral folder  :

hive@mr3worker-0cd9-18:/opt/mr3-run/hive$ ls -ll /ephemeral

total 8

drwxrwxrwx 1 root root 4096 Nov 10  2020 mr3-run

drwxrwxrwx 1 root root 4096 Nov 10  2020 temp

hive@mr3worker-0cd9-18:/opt/mr3-run/hive$

hive@mr3worker-0cd9-18:/opt/mr3-run/hive$

hive@mr3worker-0cd9-18:/opt/mr3-run/hive$ ls -ll /ephemeral/temp

total 0

hive@mr3worker-0cd9-18:/opt/mr3-run/hive$ ls -ll /ephemeral/mr3-run/

total 8

drwxrwxrwx 1 root root 4096 Nov 10  2020 buffer-dir

drwxrwxrwx 1 root root 4096 Nov 10  2020 work-dir

hive@mr3worker-0cd9-18:/opt/mr3-run/hive$ ls -ll /ephemeral/mr3-run/work-dir/

total 0

hive@mr3worker-0cd9-18:/opt/mr3-run/hive$ ls -ll /ephemeral/mr3-run/buffer-dir/

total 0

fyi : I have following values in values.yaml

workDir:

  # if false, the user should manually create PersistentVolumeClaim of the same name specified by the 

  # field name.persistentVolume, and PersistentVolume if necessary

  create: false

  isNfs: false

  nfs:

    server: "10.1.91.17"

    path: "/work/nfs/hive"

  volumeSize: 10Gi

  volumeClaimSize: 10Gi

  storageClassName: ""

  # if isNfs == false, configure a volume like below.

  # volumeStr should be indented properly with a double space (or multiple double spaces) after '\n'.

  #   ex. okay  -> volumeStr: "hostPath:\n  path: /work/nfs/mr3-run-work-dir"

  #   ex. wrong -> volumeStr: "hostPath:\n path: /work/nfs/mr3-run-work-dir"

  volumeStr: "hostPath:\n  path: /data1/gla/work-dir"

metastore:

  # if create.metastore == true, this field will be ignored and

  # replaced with hivemr3-metastore-0.metastore.{{namespace}}.svc.cluster.local

  # host: hivemr3-metastore-0.metastore.datalake.svc.cluster.local

  port: 9850

  databaseName: hive5mr3

  databaseServiceName: postgres-postgresql

  # databaseHost: postgres-postgresql.datalake.svc.cluster.local

  warehouseDir: s3a://<bucker-name>/work-dir/warehouse

  dbType: postgres

  initSchema: true

  # if true, workDir/lib will be added to metastore classpath

  mountLib: false

  hostLib: false

  hostLibDir: "/home/ec2-user/lib"

  secureMode: false

  kerberosPrincipal: hive/red0@RED

  # keytab filen under dir.keytab

  kerberosKeytab: "hive.service.keytab"

[Sungwoo Park]

As mr3.k8s.pod.worker.hostpaths is set to /ephemeral, ContainerWorker Pods will try to create intermediate files like:

/ephemeral/dag_5/container_K@9/vertex_1/attempt_37982451_0000_5_01_000000_0_10003_0/file.out

The error is generated because ContainerWorker Pods cannot create such a file under /ephemeral directory for some reason.

Not sure why the problem occurs in your case, but I guess it could be:

1) a permission problem (you could check if other queries fail for the same reason), or 

2) a problem with the remaining space when the query generates huge intermediate files (you could check if small queries succeed).

The current contents of /ephemeral are not relevant.

Cheers,

--- Sungwoo

To view this discussion on the web visit https://groups.google.com/d/msgid/hive-mr3/df50ed82-3838-457f-9145-0bbdf273adfbn%40googlegroups.com.

[pk.4...@gmail.com]

1. Table has hardly 100 rows.

2. The printed message doest not include /ephemeral/dag_5/container_K@9/vertex_.... but it shows `dag_5/container_K@9/vertex_....`  . Is there a setting on mr3, I am missing ? Since other folders have been created, I am hoping (will check soon) it should be able to create `dag_5/container_K@9/vertex_` under /ephemeral too. Correct? 

3. Where should check in the log if it is access related issue? Does it print on start of mr3worker that it fails to create a folder on mounted hostpath? `/ephemeral has root access only.

[Sungwoo Park]

2. The printed message should not include /ephemeral, so it is okay. Here is an example of the overall flow.

Suppose we set mr3.k8s.pod.worker.hostpaths to /data1/k8s:

<property>
  <name>mr3.k8s.pod.worker.hostpaths</name>
  <value>/data1/k8s</value>
</property>

Inside ContainerWorker Pods, environment variable LOCAL_DIRS is set to the value of mr3.k8s.pod.worker.hostpaths.

hive@mr3worker-e207-10:/opt/mr3-run/hive$ printenv | grep LOCAL_DIRS
LOCAL_DIRS=/data1/k8s

While a query is running, we see lots of intermediate files created under /data1/k8s, e.g:

hive@mr3worker-e207-10:/opt/mr3-run/hive$ find /data1/k8s
/data1/k8s
/data1/k8s/dag_10
/data1/k8s/dag_10/container_K@10
/data1/k8s/dag_10/container_K@10/vertex_2
/data1/k8s/dag_10/container_K@10/vertex_2/attempt_5380_0000_10_02_000002_0_10453_0
/data1/k8s/dag_10/container_K@10/vertex_2/attempt_5380_0000_10_02_000005_0_10457_0
/data1/k8s/dag_10/container_K@10/vertex_2/attempt_5380_0000_10_02_000002_0_10453
/data1/k8s/dag_10/container_K@10/vertex_2/attempt_5380_0000_10_02_000002_0_10453/file.out.index
/data1/k8s/dag_10/container_K@10/vertex_2/attempt_5380_0000_10_02_000002_0_10453/file.out
...

The error message does not include /ephemeral because we use LocalDirAllocator of Hadoop. In your example, LocalDirAllocator is created with '/ephemeral' (which is the value in environment variable LOCAL_DIRS), so it already knows where to create intermediate files. Thus there is no other setting in MR3, and the error message is all we can get. The code is found at:

https://github.com/mr3project/tez-mr3/blob/master/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/task/local/output/TezTaskOutputFiles.java#L211-L219

3. If /ephemeral has root access only, ContainerWorker should run as root. You can check the ownership of the directory and who is running ContainerWorker. In my example, it is owned by user 'hive', and ContainerWorker is run as the same user 'hive'.

hive@mr3worker-e207-10:/opt/mr3-run/hive$ ls -alt /data1/k8s
total 84
drwxrwxrwx 3 hive hive 73728 Sep 20 00:01 .
drwxr-xr-x 3 root root  4096 Sep 19 23:51 ..
drwxr-xr-x 2 hive hive  4096 Aug 30 12:21 work

hive@mr3worker-e207-10:/opt/mr3-run/hive$ ps fux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
hive      5850  0.0  0.0   5748  3628 pts/0    Ss   00:01   0:00 /bin/bash
hive      5867  0.0  0.0   9388  3064 pts/0    R+   00:06   0:00  \_ ps fux
hive         1  0.0  0.0   2400   756 ?        Ss   Sep19   0:00 /bin/sh -c /opt/mr3-run/hive/run-worker.sh -XX:+AlwaysPreTouch -Xss512k -XX:+UseG1GC -XX:TLABS
hive        10 12.5  2.2 14397400 1472796 ?    Sl   Sep19   1:51 /usr/local/openjdk-8/bin/java -Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.
hive        11  0.0  0.0   5616  3136 ?        S    Sep19   0:00 /bin/bash /opt/mr3-run/hive/run-worker.sh -XX:+AlwaysPreTouch -Xss512k -XX:+UseG1GC -XX:TLABSi
hive        36 24.9 45.3 47581972 29692396 ?   Sl   Sep19   3:42  \_ /usr/local/openjdk-8/bin/java -Djavax.security.auth.useSubjectCredsOnly=false -Djava.secur

For running as a non-root user, please see this page in MR3docs:

https://mr3docs.datamonad.com/docs/k8s/advanced/run-non-root/

For the introduction to using hostPath volumes, please see 'hostPath volumes ...' in Basic Guide:

https://mr3docs.datamonad.com/docs/k8s/advanced/run-non-root/

Cheers,

--- Sungwoo

To view this discussion on the web visit https://groups.google.com/d/msgid/hive-mr3/1f2f12ef-d0bc-446c-87ee-dfdc2cd9a566n%40googlegroups.com.

[pk.4...@gmail.com]

Hello Sungwoo,

I am making some progress. The issue is `/ephemeral` has root access only and container is run as a `hive` user (as you mentioned in #3 above). The worker was throwing WARN as seen below : 

----------------

2021-09-21T03:37:05,491  INFO [main] ContainerWorker: Starting ContainerWorker as user hive with java.io.tmpdir=/opt/mr3-run/hive/tmp

2021-09-21T03:37:06,973  INFO [main] common.Utils$: Asynchronous logging enabled with k8s-mr3-container-log4j2.properties

2021-09-21T03:37:07,195  INFO [main] mr3.MR3Runtime$: Initializing MR3Runtime for tez

2021-09-21T03:37:07,577  WARN [main] fs.LocalDirAllocator$AllocatorPerContext: /ephemeral is not writable

org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not writable: /ephemeral

at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:167) ~[hadoop-common-3.1.2.jar:?]

----------------

I followed the help provided in the doc (which you had fixed it for us last year as preBootstrapCommands were not available in kops by using initContainer commands in mr3-site.xml).

Now, I have following in mr3-site.xml (I added init.container.command) as per doc:

-----------------

<property>
<name>mr3.k8s.pod.worker.hostpaths</name>
<value>/ephemeral</value>
</property>

<property>

<name>mr3.k8s.pod.worker.init.container.command</name>
<value>chown 1000:1000 /ephemeral/; ls -alt /ephemeral </value>
</property>

-------------------------------------

Adding `init.container.command` started throwing exception invalid: spec.initContainers[0].image in worker as seen below:

------------------

2021-09-21T04:44:19,996  INFO [All-In-One] AMContainer: Command for launching ContainerWorker: /opt/mr3-run/hive/run-worker.sh -XX:+AlwaysPreTouch -Xss512k -XX:+UseG1GC -XX:TLABSize=8m -XX:+ResizeTLAB -XX:+UseNUMA -XX:+AggressiveOpts -XX:InitiatingHeapOccupancyPercent=40 -XX:G1ReservePercent=20 -XX:MaxGCPauseMillis=200 -XX:MetaspaceSize=1024m -server -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -Dlog4j.configurationFile=k8s-mr3-container-log4j2.properties  -Dmr3.root.logger=INFO -Xmx2867m -Djava.io.tmpdir=/opt/mr3-run/hive/tmp com.datamonad.mr3.worker.ContainerWorker 100.97.239.199 36991 K@34369 application_25313393_0000 0 false

2021-09-21T04:44:19,996  INFO [All-In-One] ContainerView: ContainerView[34369] created AMContainer: K@34369 ContainerReuseConfig(CrossDAGContainerReuse,true)

2021-09-21T04:44:19,998  INFO [PodStateUpdater] K8sContainerLauncher: Pod added: mr3worker-409f-34369 21935418

2021-09-21T04:44:20,001 ERROR [K8sContainerLauncher] K8sContainerLauncher: Launching ContainerWorker Pod failed: K@34369

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://kubernetes.default.svc/api/v1/namespaces/datalake/pods. Message: Pod "mr3worker-409f-34369" is invalid: spec.initContainers[0].image: Required value. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.initContainers[0].image, message=Required value, reason=FieldValueRequired, additionalProperties={})], group=null, kind=Pod, name=mr3worker-409f-34369, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Pod "mr3worker-409f-34369" is invalid: spec.initContainers[0].image: Required value, metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).

at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:503) ~[mr3-tez-1.0-assembly.jar:1.0]

at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:442) ~[mr3-tez-1.0-assembly.jar:1.0]

at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:406) ~[mr3-tez-1.0-assembly.jar:1.0]

at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleRes

------------------

Let me know how to apply initContainer commands to update the ownership or permission of those directories for hostPath volumes. Thanks for the support. Much appreciated.

Best,

- Praveen

[Sungwoo Park]

Hello,

It's probably because mr3.k8s.pod.worker.init.container.image is not specified. This configuration key specifies the Docker image for executing initContainers. You need to make sure that the Docker image can be downloaded without limitations (i.e., as many times as possible), since every ContainerWorker Pods download the Docker image.

Example:

<property>
  <name>mr3.k8s.pod.worker.init.container.image</name>
  <value>busybox</value>
</property>

Cheers,

--- Sungwoo

To view this discussion on the web visit https://groups.google.com/d/msgid/hive-mr3/83471953-bbd5-4413-9716-66603b20f39en%40googlegroups.com.

[Praveen Kumar]

Hi Sungwoo,

Thank you. That was the issue. MR3 doc : https://mr3docs.datamonad.com/docs/k8s/advanced/run-non-root/ needs that missing info, if you agree.

Thanks a ton. Much appreciated. Will keep you posted on our progress/development.

In the next few weeks I will use the MR3 latest release.

Best,

- Praveen