Composer with airflow 1.10.1 no longer has boto3 installed

[rich.m...@cloudtp.com]

We've just changed from Composer with an Airflow version of 1.10.0 to Composer with an airflow version of 1.10.1.

We are using thsse S3ToGoogleCloudStorageOperator which uses the AWS_Hook.

It appears that Composer with airflow 1.10.1 no longer deploys the boto3 python package. It was deployed in the previous version.

Is this an oversight? I haven't seen it mentioned anywhere.

Thanks,

Rich

[rajashekar...@figmd.com]

hi,

may be composer we are using in GCP...they might think we will use gs bucket instead of AWS s3 bucket...me also faced same problem...i installed external boto3 package.

Thanks,

Rajashekar

[Shehzan Devani]

Hi,

In Composer 1.5.0, the airflow installation stopped including the "devel" extras. Boto3 was one of the packages that got installed as a part of "devel" which is why it is no longer included. I would recommend adding installing boto3 using the Composer UI.

[rich.m...@cloudtp.com]

Thanks for your reply! 

Do you have a list of the "devel" extras - or a link to it? Just to cross check what we are using. 

[Shehzan Devani]

Sure, it's in setup.py. You'll notice boto3 isn't listed there, but it's actually a dependency of moto. To get a full list, you would probably want to install each of those in a virtualenv and run "pip freeze" or use pipdeptree.

The fool-proof method would be to explicitly add all dependencies that you need in the composer UI.

The information contained in this e-mail message may be privileged and confidential information and is intended only for the use of the individual and/or entity identified in the alias address of this message. If the reader of this message is not the intended recipient, or an employee or agent responsible to deliver it to the intended recipient, you are hereby notified that you have received this in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this message in error, please notify us immediately by telephone or return e-mail and delete the original message from your system.

--
You received this message because you are subscribed to a topic in the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cloud-composer-discuss/C_T5foGX1G0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/e61f9b4a-1eed-4103-9f74-319f10069d37%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Shehzan Devani

Software Engineer @ Google

972.408.7465

[rich.m...@cloudtp.com]

Thanks for your help! Really appreciate it and answers our questions.

We were looking in there.. so previously was installing just the 'devel' subpackage from airflow (pip install apache-airflow[devel]
So we can get an idea from that.

And as of 1.5.0 you are only installing these packages (and dependencies): https://cloud.google.com/composer/docs/concepts/versioning/composer-versions

(added as reference for others )

On Tuesday, February 26, 2019 at 1:26:19 PM UTC-5, Shehzan Devani wrote:

Sure, it's in setup.py. You'll notice boto3 isn't listed there, but it's actually a dependency of moto. To get a full list, you would probably want to install each of those in a virtualenv and run "pip freeze" or use pipdeptree.

The fool-proof method would be to explicitly add all dependencies that you need in the composer UI.

On Tue, Feb 26, 2019 at 10:03 AM <rich.m...@cloudtp.com> wrote:

Thanks for your reply! 

Do you have a list of the "devel" extras - or a link to it? Just to cross check what we are using. 

On Tuesday, February 26, 2019 at 12:08:42 PM UTC-5, Shehzan Devani wrote:

Hi,

In Composer 1.5.0, the airflow installation stopped including the "devel" extras. Boto3 was one of the packages that got installed as a part of "devel" which is why it is no longer included. I would recommend adding installing boto3 using the Composer UI.

On Monday, February 25, 2019 at 2:02:57 PM UTC-8, rich.m...@cloudtp.com wrote:

We've just changed from Composer with an Airflow version of 1.10.0 to Composer with an airflow version of 1.10.1.

We are using thsse S3ToGoogleCloudStorageOperator which uses the AWS_Hook.

It appears that Composer with airflow 1.10.1 no longer deploys the boto3 python package. It was deployed in the previous version.

Is this an oversight? I haven't seen it mentioned anywhere.

Thanks,

Rich

The information contained in this e-mail message may be privileged and confidential information and is intended only for the use of the individual and/or entity identified in the alias address of this message. If the reader of this message is not the intended recipient, or an employee or agent responsible to deliver it to the intended recipient, you are hereby notified that you have received this in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this message in error, please notify us immediately by telephone or return e-mail and delete the original message from your system.

--
You received this message because you are subscribed to a topic in the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cloud-composer-discuss/C_T5foGX1G0/unsubscribe.

To unsubscribe from this group and all its topics, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/e61f9b4a-1eed-4103-9f74-319f10069d37%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.