S3 Read Write Limitation Issue (Service: Amazon S3; Status Code: 503; Error Code: 503 Slow Down)

[Akash Gupta]

When running with larger machines in a cluster (for example 17 m5.4xlarge with 16 cores, 64GB memory) sometimes this error was received when writing to the table:

com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Slow Down (Service: Amazon S3; Status Code: 503; Error Code: 503 Slow Down)

[Tathagata Das]

This is essentially S3's rate limiting. It happens when you are generating too many files/objects too fast and S3 is not happy with it. How to solve it depends on your workload, either slow down the data write by using a smaller cluster or tune your workloads parameters to generate larger files (therefore, a lesser number of objects for the same write size).

On Thu, Apr 16, 2020 at 3:09 AM Akash Gupta <akash...@gmail.com> wrote:

When running with larger machines in a cluster (for example 17 m5.4xlarge with 16 cores, 64GB memory) sometimes this error was received when writing to the table:

com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Slow Down (Service: Amazon S3; Status Code: 503; Error Code: 503 Slow Down)

--
You received this message because you are subscribed to the Google Groups "Delta Lake Users and Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to delta-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/e431a4dc-5454-4273-a0c7-0669824df932%40googlegroups.com.

[Jin Lee]

AWS recommends some strategy/workaround related to this:

https://aws.amazon.com/premiumsupport/knowledge-center/emr-s3-503-slow-down/

To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/CA%2BAHuKmRmjPw3YL-Fe64Hqp%2BJB_5UgroHcKZZ1-bxEuVkbw_WQ%40mail.gmail.com.

--

Jin Lee

Architect - Data, AI, Infrastructure

doc.ai

AI -  federated learning - on real world data to develop predictive models and personal health insights.

The information contained in this e-mail is for the exclusive use of the intended recipient(s) and may be confidential, proprietary, and/or legally privileged.  Inadvertent disclosure of this message does not constitute a waiver of any privilege.  If you receive this message in error, please do not directly or indirectly use, print, copy, forward, or disclose any part of this message.  Please also delete this e-mail and all copies and notify the sender.  Thank you.

[Sanchit Gautam]

Having a micro-batching strategy in your processing is also important along with the larger file size. Also, take into consideration the Object-level invocations taking place in your system.

To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/CAJq9cQadAHYGWrvWfQvScj7-cRLZZAz5UOPDkvpqhsMKq_YGkg%40mail.gmail.com.

--

Thanks&Regards,

Sanchit Gautam

09807344668

[Gourav Sengupta]

Hi,

one of the other methods that you could use is by enabling the EMRFS consistent view. That will massively speed up the process as well :) 

Thanks and Regards,

Gourav Sengupta 

To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/CAPjxFBBKQpZhgMds%3DCWOtQ6iWJSJDnC3N5EEKgoS7uiGBStoFA%40mail.gmail.com.