Node.js Standard: memory usage different for locally and deployed version on 30 - 40 Mb.

[mr.e...@gmail.com]

Locally I've such values:

{
  rss: 82931712,

  heapTotal: 45592576,

  heapUsed: 39920088,

  external: 43999
}

but deployed version has such values:

{ 
  rss: 125652992, 
  heapTotal: 52449280, 
  heapUsed: 35882816, 
  external: 60351
} 

Locally I tested so:

1. Build app (same step before deploying).

2. Move to directory where built app.

3. Run "npm install".

4. Run "NODE_ENV=production npm start" ("node --optimize_for_size --max_old_space_size=100 --gc_interval=100 index.js"). This command also run "Appengine" for starting app.

Because of this memory overhead I can't use smallest instance class with limit to 128Mb RAM. I often see in the logs:

Exceeded soft private memory limit of 128 MB with 129 MB after servicing 4 requests total. Consider setting a larger instance class in app.yaml.

After handling this request, the process that handled this request was found to be using too much memory and was terminated. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may have a memory leak in your application.

Why deployed version has such RAM overhead?

[mr.e...@gmail.com]

I use "MacOs", but "App Engine" uses "Linux". Maybe therefore is such different? "Node.js" version "v8.11.2" is same as on "App Engine".

[mr.e...@gmail.com]

I added to first line to start script.

console.log(process.memoryUsage());

Locally:

{
  rss: 21356544,

  heapTotal: 7159808,

  heapUsed: 4475904,

  external: 26358
}

Deployed:

{ 
  rss: 54624256, 
  heapTotal: 7159808, 
  heapUsed: 4317344, 
  external: 26358
} 

[George (Cloud Platform Support)]

What is it meant here by: "This command also run "Appengine" for starting app."? How did you deploy your app? The usual command is gcloud app deploy. You may find more related information in the "Quickstart for Node.js in the App Engine Standard Environment" online document. [1]

What is the structure of your app? One would expect: 

my-nodejs-service/

  app.yaml

  package.json

  server.js

More related information may be read on the "Writing Your Web Service with Node.js". [2]

It seems you run your app locally with parameters that differ by reference to the deployed app. 

What is the output of the "gcloud info" command in your local environment? 

[1] https://cloud.google.com/nodejs/getting-started/hello-world

[2] https://cloud.google.com/appengine/docs/standard/nodejs/building-app/writing-web-service

[mr.e...@gmail.com]

If you will know own documentation better you will understand me. But ok, I will explain to you.

if to follow on the link:

https://cloud.google.com/appengine/docs/standard/nodejs/building-app/deploying-web-service

there is phrase:

"Node.js modules are installed in the cloud as listed in your package.json and package-lock.json files and your service is started by using npm start."

That's if we will define "scripts" / "start" section in "package.json" then my service will started by this command.

I use such command to decrease use RAM and because my script called "index.js" instead of "server.js":

"scripts": {

 "start": "node --optimize_for_size --gc_interval=100 index.js"

}

But you didn't understand essence of the issue.

Ok, special for testing I created "hello-world" example without "scripts" section in "package.json" and only added to first line in "index.js" next command: "console.log(process.memoryUsage());".

When I execute it locally by next command "NODE_ENV=production npm start" I see in the console a following results:

{ rss: 20860928,

  heapTotal: 7159808,

  heapUsed: 4417056,

  external: 8224 }

But when it's deployed it logs a following results:

{ rss: 55054336, 
  heapTotal: 7159808, 
  heapUsed: 4258536, 
  external: 8224 }

Output "gcloud info":

Google Cloud SDK [204.0.0]

Platform: [Mac OS X, x86_64] ('Darwin', 'Mini-Aleksandr', '17.6.0', 'Darwin Kernel Version 17.6.0: Tue May  8 15:22:16 PDT 2018; root:xnu-4570.61.1~1/RELEASE_X86_64', 'x86_64', 'i386')

Python Version: [2.7.10 (default, Oct  6 2017, 22:29:07)  [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.31)]]

Python Location: [/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python]

Site Packages: [Disabled]

Installation Root: [/Users/aleksandr/Dev/google-cloud-sdk]

Installed Components:

  core: [2018.06.04]

  gcloud: []

  beta: [2017.09.15]

  gsutil: [4.31]

  bq: [2.0.34]

  alpha: [2017.09.15]

  cloud-datastore-emulator: [2.0.0]

System PATH: [/Users/aleksandr/Dev/google-cloud-sdk/bin:/Users/aleksandr/.cargo/bin:/Users/aleksandr/mongodb/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/aleksandr/Dev/google-cloud-sdk/bin:/Users/aleksandr/.cargo/bin:/Users/aleksandr/mongodb/bin:/Users/aleksandr/n/bin]

Python PATH: [/Users/aleksandr/Dev/google-cloud-sdk/lib/third_party:/Users/aleksandr/Dev/google-cloud-sdk/lib:/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip:/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/:/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin:/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac:/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages:/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk:/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old:/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload]

Cloud SDK on PATH: [True]

Kubectl on PATH: [False]

Installation Properties: [/Users/aleksandr/Dev/google-cloud-sdk/properties]

User Config Directory: [/Users/aleksandr/.config/gcloud]

Active Configuration Name: [default]

Active Configuration Path: [/Users/aleksandr/.config/gcloud/configurations/config_default]

Account: [my email]

Project: [my project]

Current Properties:

  [core]

    project: [my project]

    account: [my email]

    disable_usage_reporting: [True]

  [compute]

    region: [us-central1]

    zone: [us-central1-a]

Logs Directory: [/Users/aleksandr/.config/gcloud/logs]

Last Log File: [/Users/aleksandr/.config/gcloud/logs/2018.06.14/09.17.51.810561.log]

git: [git version 2.13.0]

ssh: [OpenSSH_7.6p1, LibreSSL 2.6.2]

[George (Cloud Platform Support)]

We'll examine the wealth of information provided and attempt reproduction of the issue. Meanwhile, your patience is appreciated. 

[mr.e...@gmail.com]

ok, thanks

[George (Cloud Platform Support)]

Your issue warrants a more in-depth investigation. To facilitate communication and tracking of progress, we have opened this issue in the Public Issue Tracker. You may monitor it for further developments. There is no estimated time to resolution as yet. 

[Aleksander Efremov]

Ok. Will hope that will successful resolved soon.

вс, 24 июн. 2018 г., 0:59 'George (Cloud Platform Support)' via Google App Engine <google-a...@googlegroups.com>:

Your issue warrants a more in-depth investigation. To facilitate communication and tracking of progress, we have opened this issue in the Public Issue Tracker. You may monitor it for further developments. There is no estimated time to resolution as yet. 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/08c4333a-d3c6-41a0-b6db-e2df363e2abb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[George (Cloud Platform Support)]

Hello Aleksander, 

In fact there is no estimated time to resolution, but I am sure Developers will address the issue as soon as possible, taken into account their priorities. 

[Aleksander Efremov]

Hi George.

That's ok. The main thing there is created issue. Thanks for helping.

[George (Cloud Platform Support)]

Hi Aleksander, 

The issue should be fixed by now; you are encouraged to test it again. 

[Aleksander Efremov]

Yes, it's better now, thanks. Before was difference from my PC and Cloud Platform about 20Mb, now 10Mb.

[George (Cloud Platform Support)]

Hello Aleksander, 

Glad to read you find the solution working. Do you consider this solution as definitive? Would you like to see improvements? If so, what would the ideal solution mean for you? 

[Александр Ефремов]

Hi George.

For me it's enough now, but exists such saying: there is no limit to perfection -:).

I also made some optimizations therefore now I'm able to use most small instance.

But if exists opportunity to improve that, of course would be still better.

8 авг. 2018 г., в 22:41, 'George (Cloud Platform Support)' via Google App Engine <google-a...@googlegroups.com> написал(а):

Hello Aleksander, 

Glad to read you find the solution working. Do you consider this solution as definitive? Would you like to see improvements? If so, what would the ideal solution mean for you? 

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/fa9etnYlog0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.

To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/5e8763dc-76d9-445f-8b35-8066d6e25d28%40googlegroups.com.

[George (Cloud Platform Support)]

Hi Aleksander, 

We agree without doubt that in an ideal world reaching perfection is a realist target. Even in our real world, I am sure Developers would like to fine-tune all products to perfection, but in a real world time is an issue. There are serious issues to address and to fix, whose urgency has to be met with timely results. As a consequence, in your particular case, if you indicate solution works and is enough for you, it should come as no surprise that Engineering would rather concentrate on outages and urgent fixes to be implemented right now. Your understanding is much appreciated. 

There is an unexpected result of your optimizations: how much did your changes contribute to improvement, vs. the Engineering solution; difficult to judge. 

[Aleksander Efremov]

I won't argue, it's enough for me. Thanks for helping.

[Александр Ефремов]

But also exists same problem with Cloud Functions. There is created the same issue?

> 10 авг. 2018 г., в 7:32, Aleksander Efremov <mr.e...@gmail.com> написал(а):

>
> I won't argue, it's enough for me. Thanks for helping.
>

> --
> You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/fa9etnYlog0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
> To post to this group, send email to google-a...@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.

> To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/ff3ca4a0-e7fa-44f8-b231-fa78e1ad77b3%40googlegroups.com.

[George (Cloud Platform Support)]

Hi Aleksander, 

If you speak about a different, even if somehow related issue, it is better to open a separate discussion on that subject. As this Cloud Functions problem looks more like a technical issue, rather than general or opinion-based, and needs fixing, it is better to open it in the Public Issue Tracker. It can get tracked easier there, and you may follow progress towards an eventual solution. 

[Aleksander Efremov]

Hi George.

Thanks, I will try to follow your offer.

пт, 10 авг. 2018 г., 21:52 'George (Cloud Platform Support)' via Google App Engine <google-a...@googlegroups.com>:

Hi Aleksander, 

If you speak about a different, even if somehow related issue, it is better to open a separate discussion on that subject. As this Cloud Functions problem looks more like a technical issue, rather than general or opinion-based, and needs fixing, it is better to open it in the Public Issue Tracker. It can get tracked easier there, and you may follow progress towards an eventual solution. 

--

You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/fa9etnYlog0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/45b7a227-7885-4c8d-814f-c6815752cad9%40googlegroups.com.