My two cents is you have to do this via s3. It sure seems like you could go a long way on AWS free tier. I use databricks but I’ve explored doing this kind of work with AWS glue or emr. I’m convinced you can go a long way with just glue. Also have you considered AWS search? Writing a lambda function to query file nams and offsets and write them back to something seems fairly affordable.
Sent from my iPhone
> On Jun 21, 2025, at 8:47 AM,
common...@googlegroups.com wrote:
>
> But the thing is, due to a very limited budget, I am unable to use AWS for
> this, as an alternative, I consider using HTTPS range requests