KubernetesPodOperator crash if do_xcom_push is set to true

[Szymon Sikora]

I'm learning how to use KubernetesPodOperator on Cloud Composer.

I have a problems when i request for a x-com-sidecar by passing do_xcom_push argument on a Composer environment.

I receive following error:

Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: unable to set memory limit to 1073742 (current usage: 2777088, peak usage: 2990080): unknown

What may cause this error? I've already tried to increase worker count and memory (max workers = 6, memory=4Gb) but without any success.

Without xcom-side-car, KubernetesPodOperator works as expected, container starts and runs correctly.

My current composer image: composer-2.1.5-airflow-2.4.3

I'm using following config:

resource "google_composer_environment" "airflow" {

name = "composer-env"

region = var.region_name

config {

software_config {

image_version = "composer-2.1.5-airflow-2.4.3"

pypi_packages = {

apache-airflow-providers-slack = ""

}

airflow_config_overrides = {

core-dags_are_paused_at_creation = "True"

}

}

node_config {

service_account = "sa-airflow@${var.project_name}.iam.gserviceaccount.com"

}

workloads_config {

worker {

cpu = 1

memory_gb = 4

storage_gb = 4

min_count = 2

max_count = 6

}

web_server {

cpu = 0.5

memory_gb = 2

storage_gb = 1

}

scheduler {

count = 1

cpu = 1

memory_gb = 4

storage_gb = 1

}

}

}

}

[Szymon Sikora]

Side note - i'm trying to run a pod with following k8s config: 

namespace="composer-user-workloads"

kubernetes_conn_id="kubernetes_default"

config_file="/home/airflow/composer_kube_config",