Sync Google Cloud Storage to Instance Cluster

[Dennis Yurkevich]

Hello,

If I write a cron job to fetch data from Cloud Storage and write it to the instance file system - will this be replicated across all of my instances?

Thanks,

Dennis

[Amit (Google Cloud Support)]

Hello Dennis,

App Engine instances are computing units that App Engine uses to automatically scale the running application. Each instance has its own security layer to ensure that instances cannot unintentionally affect each other. In this similar standpoint, the files written to the file system on one instance will not be replicated across all other available instances. For more information about how instances are managed on App Engine, please go through this [1] article.

I hope this answers your question.

[1] https://cloud.google.com/appengine/docs/standard/python/how-instances-are-managed

[Dennis Yurkevich]

Hello Amit,

Appreciate the reply, and thank you for the resource.

However - my question is still open, at the very basic level I would like to read an external data source and load an object to be available in memory on all of my instances.

I am using nodejs flexible and memcache is not available, or ideal as I need the data to persist. 

Ideally I could maybe mount the google cloud bucket as a filesystem, and run a CRON job / task to load any new files from that bucket into memory. 

Would be great if you can suggest an approach here.

Best,

Dennis

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/d2c09cca-f7d5-4810-9b7e-d59e047c0cfd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Attila-Mihaly Balazs]

You can take a look at Google Datastore or Google Cloud Storage (perhaps with some Pub/Sub thrown in to notify live instances that they need to load/reload their data).

Attila

[Dennis Yurkevich]

This answer does not help much.

I am trying to understand what the mechanics are for adding data to memory where the data originates from does not matter - I am able to see the various google products available.

Why does pub/sub load data into both instances and cron job only into one?

On Fri, Aug 24, 2018 at 6:25 AM, Attila-Mihaly Balazs <dify...@gmail.com> wrote:

You can take a look at Google Datastore or Google Cloud Storage (perhaps with some Pub/Sub thrown in to notify live instances that they need to load/reload their data).

Attila

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/a3945ade-d570-49f1-b009-91721732f7d6%40googlegroups.com.

[Attila-Mihaly Balazs]

Pub/sub can help you to notify all running instances that new data is available and thus they can re-fetch the data as soon as it's updated rather than having to check periodically for new data.

Cheers,

Attila

[Amit (Google Cloud Support)]

Hello Dennis,

Yes, you are right. The Memcache service is not available for App Engine flexible environment at this moment. However, an alpha version will be available soon, you can fill-out the early access form here. Currently , you can use Redis Labs Memcache, a third party fully managed service, to cache application data. This will allow you to use Memcached in order to achieve the persistence and in-memory replication of your data.

Alternatively, you can use Node.JS Cloud Storage client library to access your files on Cloud Storage.

I hope that the provided will helps you to find a resolution for your issue.

[Dennis Yurkevich]

Hello Amit,

I am capable of locating the documentation myself. I was under the impression that this mailing list is to discuss GAE as a product.

I have a use case which I do not feel is well satisfied by what is available in the docs/product itself.

Dennis

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/19436222-7877-4a3e-867d-0b5a2c573b02%40googlegroups.com.

[Katayoon (Cloud Platform Support)]

I should add that Google Groups are reserved for general product discussions. We cannot provide you any advice on your system’s architecture. If you have a Premium support package, the architecture advisory service is available on best-effort basis or you may refer to the Google Cloud Partners.  

Furthermore, you can send your feature request via the Issue Tracker if any so that the product team will evaluate your request.

[Dennis Yurkevich]

Cloud storage being accessed by a Cloud VM is quite general in my view.

On Sun, Sep 2, 2018 at 5:26 PM, 'Katayoon (Cloud Platform Support)' via Google App Engine <google-a...@googlegroups.com> wrote:

I should add that Google Groups are reserved for general product discussions. We cannot provide you any advice on your system’s architecture. If you have a Premium support package, the architecture advisory service is available on best-effort basis or you may refer to the Google Cloud Partners.  

Furthermore, you can send your feature request via the Issue Tracker if any so that the product team will evaluate your request.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/cb9f302d-3a99-495e-aac9-98a6e16c3c66%40googlegroups.com.

[David (Google Cloud Support)]

Good suggestions about how to replicate data across all of the instances were given. If you don’t feel satisfied with these, I believe that if you elaborate more on your use case and explain what you are specifically trying to accomplish (why do you want to replicate data across all of your instances) you might get more suggestions, but you might need to do this in Stack Overflow as it would be out of the general scope and by posting your question on Stack Overflow, you will gain access to a large number of experts as well.

Alternatively, as Katayoon previously mentioned, you can file a feature request via Issue Tracker and the App Engine Team will evaluate your request.