PV/C Issues When Kubernets with Helm

[David Engel]

The removal of the typescript method for running MR3 on Kubernetes
caught us out when the mr3project/hive:4.0.0.mr3.2.4 image was
replaced by mr3project/hive:4.0.0.mr3.2.6 on Docker Hub. The new
image reported some strange errors when we tried to re-use our old
typescript configuration with it. As a result, we temporarily
restored the old image from Docker cache while we try to convert to
one of the still supported methods for K8s.

The Helm method seems to be the cleaner method so that's the method we
are trying first. It kind of works but fails sometimes/eventually
with the PV/PVC issues described below.

1. The workdir-pvc doesn't get bound sometimes. We have not been able
to figure out yet why this happens. A restart or two usually clears
this issue up.

2. We have two custom PV/PVCs that we use to mount /home and /sta
directories into the metastore, hiveserver2 and worker pods. For
reasons that we totally don't understand, these two mounts get
reversed sometimes. By reversed, we mean that /sta gets mounted on
/home and vice versa. Like with the workdir issue, this usually gets
cleared up with a restart or two.

While annoying, these two issues are not blockers as they can be
quickly checked for after startup and get resolved with another
restart or two.

3. The /sta and /home PV/PVCs stop working after a while in the worker
pods as new pods get created.

This last issue is blocker as we have external tables and transform
scripts in those directories that stop working when they aren't
mounted.

Please let us know what information you would like to see to help
diagnose these issues. We can schedule some maintenance windows next
week to gather any desired logs. We can also provide our
customizations to the mr3 git repo for using Helm.

David
--
David Engel
da...@istwok.net

[Sungwoo Park]

hive:4.0.0.mr3.2.6 is a temporary image created with a release candidate of MR3 2.6.

There are lots of changes in MR3 2.6 compared with MR3 2.4, but almost all of them are about performance optimizations, and we did not make any changes specific to Kubernetes and PV/PVC.

I would suggest continuing to use the last working image and scripts (Helm or TypeScript), as we still need more time to release MR3 2.6.

--- Sungwoo

--
You received this message because you are subscribed to the Google Groups "MR3" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hive-mr3+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/hive-mr3/aj7TOb8nkulycz3U%40opus.istwok.net.

[David Engel]

Okay. FYI, though, the whole reason we started down this path was
because hive:4.0.0.mr3.2.4 was removed or expired on Docker Hub.
hive:4.0.0.mr3.2.6 is still the only one there now. We had to
retrieve hive:4.0.0.mr3.2.4 from local cache and upload it to our own
repo to keep using it.

David

--
David Engel
da...@istwok.net