composer-2.0.0-preview.5-airflow-2.1.4 - Worker Disk - SFTPToGCSOperator

[Bhaumik Shah]

Hi, 

We are currently using  SFTPToGCSOperator and having issues pulling large files from SFTP and pushing them to GCS. For instances with Composer 1.0, we have specified a larger disk size to accommodate large files transfer. Composer 2.0 limits worker disk size to 10GB and seems like that's not working with existing dags and breaking large files SFTP transfer. 

We are in process of moving SFTPtoGCS copy outside of the Airflow pipeline, but for now, checking if any other suggestions/ workarounds.

Thanks,

Bhaumik