Another disappointment with Cloud Composer

[sgus...@butterflynetinc.com]

Set of packages which was working on composer-1.11.0-airflow-1.10.6 does not work on anymore in composer-1.12.2-airflow-1.10.6, some required downgrade and it's not possible to upgrade them as it failing with Build issue (none of these two packages required by my tasks)

Warning!!! Possibly conflicting dependencies found:

* google-cloud-pubsub==1.6.1

- google-api-core [required: >=1.14.0,!=1.19.*,!=1.18.*,!=1.17.*, installed: 1.17.0]

* apache-beam==2.20.0

- httplib2 [required: >=0.8,<=0.12.0, installed: 0.13.1]

- oauth2client [required: >=2.0.1,<4, installed: 4.1.3]

it's unbelievably hard to work with composer.

Sergei

[Rafal Biegacz]

Hi Sergei,

Please, take a look a this release notes

One of the feature requests coming from Composer users was to have a better control over situations when user-installed PyPi modules generate conflicts of Python modules. As a result of such conflicts, sometimes one could make other already installed PyPi modules stop working.

Now you have a control over whether you want to avoid PyPi module conflicts or you want to allow for them.

If you set IGNORE_PYPI_DEPENDENCY_CONFLICTS environment variable to True, it means that you are aware of the potential conflicts and nevertheless you want to proceed with Python module installations.

Regards, Rafal.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/69cf2125-7a1d-43f1-b3f6-4b48e5a9febbn%40googlegroups.com.

[sgus...@butterflynetinc.com]

Rafal,

My main issue with Composer as every new release add some kind of troubles instead of solving them.

Look like this need to be set anyway as i can't see a way to go around:

There are incompatible versions in the resolved dependencies:

  pyarrow<0.18.0,>=0.15.1 (from apache-beam==2.24.0->-r dag_requirements.in (line 2))

  pyarrow<2.0dev,>=1.0.0 (from google-cloud-bigquery[bqstorage,pandas]==2.1.0->-r dag_requirements.in (line 12))

I don't use apache-beam and i put latest version and still can't get around BigQuery requirements.

Sergei

[ole.christia...@unacast.com]

Sergei, have you tried running locally and specifying version list according to package list from from https://cloud.google.com/composer/docs/concepts/versioning/composer-versions

I've also experienced a few surprises during upgrades, so are now using the version list from that url and testing before upgrades. Testing/running locally in a docker container built on python:3.6-stretch and installing gcloud as well.

//OC

[sgus...@butterflynetinc.com]

Unfortunately, some of the packages which i have require higher version, for example Composer ok with oauth2client version 3.0.0 which was released 4 years ago.
I will work with version when i will be doing upgrade to Airflow 1.10.10 which i am eager to do.

Sergei

[Ole Christian Langfjæran]

Is PythonVirtualEnvOperator an alternative perhaps? Or KubernetesPodOperator/GKEOperator?

//OC

[sgus...@butterflynetinc.com]

i think KubernetesPodOperator will be overkill for my needs, i need to take a look into PythonVirtualEnvOperator (my thoughts that will add latency to tasks).
I wish the Airflow to properly support pools and each can be configured with set of requirements and different machines sizes.

Sergei