Best solution / practice for temp files with App Engine PHP?

[Alex Kerr]

Hi,

What's generally accepted wisdom/tech solution for temp files on GAE using PHP, please?

I need to store and process an image (a few hundred K in size) using PHP that momentarily (e.g. < 1 second) needs to exist as a temp file somewhere before it's sent on elsewhere and the temp file can be deleted. My site will need to autoscale for potentially large numbers of users (using GAE as standard) - i.e. so site may need to be doing this for many users simultaneously (all for different images) on however many GAE instances are running.

My idea was to attempt to store the temp file in memory (using tempnam() etc) and if that failed (e.g. mem full on that instance), immediately try and use some other storage instead on the fly. Question is what? Image has to be available as a file for CURL to access it (I think?) and send elsewhere, so Memcache is not an option (can't access the data as a file to pass into CURL - or can I?), but e.g. Cloud Storage is (via 'gs://[bucket-name]/...'). Thing is, if I've just written the file to storage, is it immediately available for reading? That's also a significant cost incurred... Or Cloud Datastore? But again is file available for reading immediately, and is there a file wrapper for access...?

Any help much appreciated!

Thanks,

Alex

[Stuart Langley]

Have you tried using tempnam? If so, what problems did you see? That seems to be the obvious solution.

[Alex Kerr]

Thanks. No problems with tempnam, but I'm capacity planning for what is likely to be a heavily loaded site with many users hitting simultaneously. I want to avoid (or plan for) a situation where there is no spare memory in a particular instance for another temp file to be created because it's serving so many other users at the same time. I'm not sure if this is a likely scenario or not because I don't know what the potential temp folder capacity is and the maximum users one instance is likely to be serving simultaneously.

Current strategy is to try and create a temp file in vfs:// and if that fails create one in Cloud Storage - but is it suitable for that kind of momentary usage and would I incur significant cost (I realise no cost for bandwidth if in the same region) or what's the minimum cost for each momentary usage of Cloud Storage - is it billed by the second, hour or day? (i.e. if used for 1 second, I get billed for a day?)

Many thanks,

Alex

[Mars Lan]

You can find the pricing details for GCS at https://cloud.google.com/storage/pricing

[Stuart Langley]

If you're doing images a few hundred K in size I seriously doubt you'll hit memory limits using vfs:// - I'd start there.

[Alex Kerr]

Yes, the pricing details were my first port of call :)

But they say nothing about costs accrued for data stored for very short periods of time...

[Alex Kerr]

Thanks Stuart, good to have that reassurance. My code does do that, but in the event a file can't be created in vfs:// it saves it out to Cloud Storage. I can't find details anywhere on how soon after a file is written to Cloud Storage it becomes available for reading again, and also what costs would be accured for momentarily stored files (I'd be deleting it again straight away), do you happen to know anything about those?

Many thanks,

Alex

[Stuart Langley]

Cloud storage read-after-write is strongly consistent so the file would be available immediately after it is written: https://cloud.google.com/storage/docs/concepts-techniques#consistency

I'm not certain about the costs for "temporary" files - If the files are 100kb and the cost per GB per month is $0.026 then I can't imagine it would be much. Plus you have the default bucket for the app which is 5GB for free from memory.

[Alex Kerr]

That's helpful thanks. Only problem is I deleted the default bucket using the cloud console, is it possible to recreate it? (I tried and it's asking me to verify the bucket name). Or do any buckets under a project count for the 5 GB free?

[Stuart Langley]

No way to create it - you'll need a new app instead.