Copy, Post and List Requests exceeded Quota but no data stored on AWS

[Abhishek Dhadwal]

Good day  sir/madam,

My colleague and I were aiming to utilize the Sensus App with AWS S3 enabled in order to run a week-long study.

While running demo studies in order to check data usage and remote storage costs, we ran into the issue of exceeding the number of the "2,000 Put, Copy, Post or List Requests of Amazon S3" quota. 

But, the bucket mentioned in the study was empty.

Q. Is this due to constant write timeouts for the magnitude of the data involved, or are there any other underlying issues at hand?

Information pertaining to the study :

- The bucket was created using the configure-s3.sh protocol.

- The following settings are used in the 'remote data store' section :

Write Delay : 3600000 ms

Write timeout : 5 minutes

Write on power connect : Yes

Write on wifi connect : Yes

Require wifi : Yes

Require charging : Yes

Required battery charge : 20 %

- 10 probes were used, with omission of device id used as the anonymization method.

- On a previous instance of testing aws storage, data did get submitted into buckets used for tests.

- Both of us had approximately 700 MB of data stored in our devices in our devices as a result of running the study for about 20 hours.

[Matthew Gerber]

Hello -

It's not clear to me why this is happening. We've run many studies with S3 without seeing this problem. Mehdi (CC'ed), any ideas?

Matt

--
You received this message because you are subscribed to the Google Groups "Sensus" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sensus-app+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sensus-app/433f7d40-21ee-4953-86cb-abf62735ad98n%40googlegroups.com.

[boukhechba, mahdi ouassim (mob3f)]

Hi,

It's not clear to me neither. We usually don't use AWS free tier and we have never seen a similar issue. Please double check your bucket, the data will be in a "subfolder" named data. Also, you can send me your log file to check if something is wrong with data upload. This can be shared by navigating to Your Studies/Log/Share on Sensus.

Best 

Mehdi

---

From: Matthew Gerber <gerber....@gmail.com>
Sent: Thursday, August 6, 2020 9:04 AM
To: Abhishek Dhadwal <dhadw...@gmail.com>
Cc: Sensus <sensu...@googlegroups.com>; boukhechba, mahdi ouassim (mob3f) <mo...@virginia.edu>
Subject: Re: Copy, Post and List Requests exceeded Quota but no data stored on AWS

 

[Abhishek Dhadwal]

Hi,

The bucket in question is empty. I've mailed the logs to you, kindly check them out.

Regards,

Abhishek

[Abhishek Dhadwal]

Update :

After enabling logging for the buckets, the following error issue is stored whenever the app tries to submit data on its own. These consisted of the majority of access requests made in the past days, hence reaching the plan quota mentioned above -

The error message (in XML) -

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>--</RequestId>
<HostId>
--</HostId>
</Error>

Are there any changes required to be made to the IAM groups the Sensus user belongs to?

PS - Making the writing permissions for the bucket public helps in transmission. But, the data gets transmitted only when I open the app, or choose the submit data option for the study.

Regards,

Abhishek

[Matthew Gerber]

Abhishek -

I haven't worked on the Sensus project in a long time, but there used to be a shell script that would set up the S3 bucket and permissions. If that's still the case, then it seems likely that the script didn't run properly. Either that, or the bucket name might have been incorrectly specified in the Sensus protocol's remote data store. Hard to say.

Matt

To view this discussion on the web visit https://groups.google.com/d/msgid/sensus-app/5d3ab00a-942d-4d31-b292-de3bf5fe4e3an%40googlegroups.com.

[Abhishek Dhadwal]

Hi,

The bucket recieves data, but only if I open the app and request to submit it.

Else, the access queries exist, but they have all been denied due to permission issues.

Yes, I had created the buckets using the script provided for RemoteDataStore.

[Matthew Gerber]

Is this problem on iOS, Android or both?

To view this discussion on the web visit https://groups.google.com/d/msgid/sensus-app/CAKUNq_GYmzXJiV%2BpGu_046yjijUvCce8vM3WychQftcZGn1Hgg%40mail.gmail.com.

[Abhishek Dhadwal]

We ran the tests in Android only, so far.

[Matthew Gerber]

On Android, the app should be able to transmit in the background. This will depend on the Android version, due to the way app wake-ups are handled. I think Mehdi (CC'ed) would need to weigh in for further assistance.

Matt

[Abhishek Dhadwal]

Update : The main (access denied) issue occurs due to the region change modifications I had made in the configure-s3.sh script.

he original script provided the following error when I attempted to configure it for the ap-south-1 region :

Command used : ./configure-s3.sh bucketname ap-south-1

Output :

"An error occurred (IllegalLocationConstraintException) when calling the CreateBucket operation: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.
Failed to create bucket."
Hence, I made a command change in the script to allow ap-south-1 access, and the bucket was ready.

The script is attached for your reference.

Would any access issues arise if I leave the bucket region as us-east-1 and the regions of participants are in South Asia?

Regards,

Abhishek

[Matthew Gerber]

If your S3 account is configured in the US east region, and the Sensus protocol is configured to submit data to it, then there shouldn't be any issues if your participants physically reside in a different region. I believe the only issue you might encounter are the latencies involved in transmitting data from your physical location to the US east region. It would be faster to transmit to an AWS region that is closer.

Matt

To view this discussion on the web visit https://groups.google.com/d/msgid/sensus-app/0cfc9e82-1aa8-4017-91f3-1e22f02fa767n%40googlegroups.com.

[Abhishek Dhadwal]

The account is configured in the ap-south-1 region, hence the aforementioned problems.

Thanks for the clarification.

Abhishek