Error [allocation failure]

31 views
Skip to first unread message

Ally Shaban

unread,
Sep 6, 2017, 5:48:17 AM9/6/17
to OpenHIM-Implementers
Hi All,
Im experiencing an error within openHIM as seen on the log below,when this error occurs it causes mediators to fail and restart while it was at the mid of performing some tasks. Is there any one with an idea on what is causing an error from line number 7 to 32?

  1. 2017-09-05T18:09:19.452Z - [32minfo [39m: [worker3] Job generate transaction alerts has completed
  2. 2017-09-05T18:09:19.453Z - [32minfo [39m: [worker3] starting job: auto retry failed transactions, Last Ran at: Tue Sep 05 201718:09:19 GMT+0000 (UTC)
  3. 2017-09-05T18:09:19.455Z - [32minfo [39m: [worker3] Job auto retry failed transactions has completed
  4.  
  5. <--- Last few GCs --->
  6.  
  7.  5895546 ms: Scavenge 50.3 (60.2) -> 49.7 (60.2) MB, 1.0 / 0.0 ms [allocation failure].
  8.  5905484 ms: Scavenge 50.5 (61.2) -> 49.9 (61.2) MB, 0.9 / 0.0 ms [allocation failure].
  9.  5905506 ms: Scavenge 50.8 (61.2) -> 50.2 (61.2) MB, 0.9 / 0.0 ms [allocation failure].
  10.  5915515 ms: Scavenge 51.1 (61.2) -> 50.4 (62.2) MB, 4.0 / 0.0 ms [allocation failure].
  11.  5924105 ms: Scavenge 51.4 (62.2) -> 50.6 (62.2) MB, 22.7 / 2.7 ms [allocation failure].
  12.  
  13.  
  14. <--- JS stacktrace --->
  15.  
  16. ==== JS stack trace =========================================
  17.  
  18. Security context: 0x27617dd3fa99 <JS Object>
  19.     2: notify [/usr/share/openhim-core/node_modules/mongoose/lib/types/documentarray.js:~228] [pc=0x394d0049a632](this=0x2952c91ffb71 <JS Array[0]>,event=0x14e195af9709 <String[4]: save>)
  20.     3: new constructor(aka MongooseDocumentArray) [/usr/share/openhim-core/node_modules/mongoose/lib/types/documentarray.js:~23][pc=0x394d012dbfbd] (this=0x2952c91ffb39 <a MongooseDocumentArray with map 0x3d...
  21.  
  22. FATAL ERROR: Committing semi space failed. Allocation failed - process out of memory
  23.  1: node::Abort() [/home/openhim/.nvm/versions/node/v6.11.2/bin/node]
  24.  2: 0x109cafc [/home/openhim/.nvm/versions/node/v6.11.2/bin/node]
  25.  3: v8::Utils::ReportApiFailure(char const*, char const*) [/home/openhim/.nvm/versions/node/v6.11.2/bin/node]
  26.  4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/home/openhim/.nvm/versions/node/v6.11.2/bin/node]
  27.  5: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags)[/home/openhim/.nvm/versions/node/v6.11.2/bin/node]
  28.  6: v8::internal::Heap::CollectGarbage(v8::internal::GarbageCollector, char const*, char const*, v8::GCCallbackFlags)[/home/openhim/.nvm/versions/node/v6.11.2/bin/node]
  29.  7: v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace)[/home/openhim/.nvm/versions/node/v6.11.2/bin/node]
  30.  8: v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*)[/home/openhim/.nvm/versions/node/v6.11.2/bin/node]
  31.  9: 0x394cff4062bf
  32. 2017-09-05T18:10:37.964Z - [33mwarn [39m: [master] worker 16217 died
  33. 2017-09-05T18:10:38.344Z - [32minfo [39m: [master] worker with pid 16950 is online
  34. Needs latest SSL Root Certificate Authority data /usr/share/openhim-core/node_modules/ssl-root-cas/ssl-root-cas-latest.js
  35. the server/replset/mongos options are deprecated, all their options are supported at the top level of the options object [poolSize,ssl,sslValidate,sslCA,sslCert,sslKey,sslPass,sslCRL,autoReconnect,noDelay,keepAlive,connectTimeoutMS,socketTimeoutMS,reconnectTries,reconnectInterval,ha,haInterval,replicaSet,secondaryAcceptableLatencyMS,acceptableLatencyMS,connectWithNoPrimary,authSource,w,wtimeout,j,forceServerObjectId,serializeFunctions,ignoreUndefined,raw,promoteLongs,bufferMaxEntries,readPreference,pkFactory,promiseLibrary,readConcern,maxStalenessSeconds,loggerLevel,logger,promoteValues,promoteBuffers,promoteLongs,domainsEnabled,keepAliveInitialDelay,checkServerIdentity,validateOptions]
  36. the server/replset/mongos options are deprecated, all their options are supported at the top level of the options object [poolSize,ssl,sslValidate,sslCA,sslCert,sslKey,sslPass,sslCRL,autoReconnect,noDelay,keepAlive,connectTimeoutMS,socketTimeoutMS,reconnectTries,reconnectInterval,ha,haInterval,replicaSet,secondaryAcceptableLatencyMS,acceptableLatencyMS,connectWithNoPrimary,authSource,w,wtimeout,j,forceServerObjectId,serializeFunctions,ignoreUndefined,raw,promoteLongs,bufferMaxEntries,readPreference,pkFactory,promiseLibrary,readConcern,maxStalenessSeconds,loggerLevel,logger,promoteValues,promoteBuffers,promoteLongs,domainsEnabled,keepAliveInitialDelay,checkServerIdentity,validateOptions]
  37. 2017-09-05T18:10:42.355Z - [32minfo [39m: [worker4] Starting OpenHIM server on 0.0.0.0...
  38.  
  39.  
  40.  
  41. Couldn't download the latest Root CAs, but it's not a big deal.
  42.  
  43. Use "require('ssl-root-cas')" instead of "require('ssl-root-cas/latest')"
  44.  
  45. 2017-09-05T18:10:42.512Z - [32minfo [39m: [worker4] Started rerun task processor
  46. 2017-09-05T18:10:42.562Z - [32minfo [39m: [worker4] HTTP listening on port 5001
  47. 2017-09-05T18:10:42.564Z - [32minfo [39m: [worker4] Transaction Rerun HTTP listening on port 7786
  48. 2017-09-05T18:10:42.565Z - [32minfo [39m: [worker4] HTTP receiver for Socket adapter listening on port 7787
  49. 2017-09-05T18:10:42.568Z - [32minfo [39m: [worker4] Polling port listening on port 7788
  50. 2017-09-05T18:10:42.698Z - [32minfo [39m: [worker4] API HTTPS listening on port 8080
  51. 2017-09-05T18:10:42.730Z - [32minfo [39m: [worker4] HTTPS listening on port 5000
  52. 2017-09-05T18:10:43.154Z - [32minfo [39m: [worker4] Starting polling server...

Ally Shaban

unread,
Sep 6, 2017, 12:01:44 PM9/6/17
to OpenHIM-Implementers
This is sorted out by upgrading node from 8.0.0 to 8.4.0

Thank you,

Ally.

Richard Langford

unread,
Sep 6, 2017, 4:01:47 PM9/6/17
to Ally Shaban, OpenHIM-Implementers
Hi Ally,

Thanks for this!

Sent from my iPhone
--
You received this message because you are subscribed to the Google Groups "OpenHIM-Implementers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openhim-implemen...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Ryan Crichton

unread,
Sep 11, 2017, 3:21:27 AM9/11/17
to Richard Langford, Ally Shaban, OpenHIM-Implementers
Hi Ally,

Sorry for the delay in response. But it also may be useful to make sure you aren't running into maxing out memory that node is allocated to use. This could cause those sorts of errors. Upgrading the node version may only hide the issue for now and it may be more memory efficient.

Cheers,
Ryan 
--
Ryan Crichton
Lead Developer, Jembi Health Systems |  SOUTH AFRICA
Mobile: +27845829934 | Skype: ryan.graham.crichton
E-mail: ry...@jembi.org

Ally Shaban

unread,
Sep 11, 2017, 6:55:03 AM9/11/17
to Ryan Crichton, Richard Langford, OpenHIM-Implementers
Hi Ryan,
You are right,It has happened again and caused my server not running. Is there any other fix that i can do apart from increasing node memory limit?

Thank you,

Ally.

To unsubscribe from this group and stop receiving emails from it, send an email to openhim-implementers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OpenHIM-Implementers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openhim-implementers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Ally Shaban

unread,
Sep 11, 2017, 8:00:47 AM9/11/17
to Ryan Crichton, Richard Langford, OpenHIM-Implementers
Hi,
I have noticed orchestration msgs are the ones causing out of memory because the openHIM API im using to update transactions does not support appending of new orchestrations to existing transaction orchestrations,this forces me to store all orchestrations until task processing ends then saving the whole orchestration msg to the transaction,so sometimes the msg can be big.

Ally.

To unsubscribe from this group and stop receiving emails from it, send an email to openhim-implementers+unsubscrib...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OpenHIM-Implementers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openhim-implementers+unsubscrib...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Ally Shaban

unread,
Sep 27, 2017, 5:34:54 AM9/27/17
to OpenHIM-Implementers
Hi All,
Is there any one with an idea on this?

Ally.

Ryan Crichton

unread,
Sep 27, 2017, 5:47:50 AM9/27/17
to Ally Shaban, OpenHIM-Implementers
Hi Ally,

There is not solution for this now. I would suggest perhaps only saving orchestration that error or providing the result of a batch of request in one orchestration to get around this when you have a massive amount of a orchestrations.

Cheers,
Ryan

To unsubscribe from this group and stop receiving emails from it, send an email to openhim-implemen...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OpenHIM-Implementers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openhim-implemen...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
--
Ryan Crichton
Lead Developer, Jembi Health Systems |  SOUTH AFRICA
Mobile: +27845829934 | Skype: ryan.graham.crichton
E-mail: ry...@jembi.org

--
You received this message because you are subscribed to the Google Groups "OpenHIM-Implementers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openhim-implemen...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Ally Shaban

unread,
Sep 27, 2017, 6:48:05 AM9/27/17
to Ryan Crichton, OpenHIM-Implementers
Hi Ryan,
I completely disabled orchestrations but still openHIM is running out of memory.

Ally.

To unsubscribe from this group and stop receiving emails from it, send an email to openhim-implementers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OpenHIM-Implementers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openhim-implementers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
--
Ryan Crichton
Lead Developer, Jembi Health Systems |  SOUTH AFRICA
Mobile: +27845829934 | Skype: ryan.graham.crichton
E-mail: ry...@jembi.org

--
You received this message because you are subscribed to the Google Groups "OpenHIM-Implementers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openhim-implementers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Ryan Crichton

unread,
Sep 27, 2017, 7:25:00 AM9/27/17
to Ally Shaban, OpenHIM-Implementers
How much memory is it using? Perhaps you need to upgrade your server? Do you know how many orchestration requests it's trying to make?

Cheers,
Ryan

To unsubscribe from this group and stop receiving emails from it, send an email to openhim-implemen...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OpenHIM-Implementers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openhim-implemen...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
--
Ryan Crichton
Lead Developer, Jembi Health Systems |  SOUTH AFRICA
Mobile: +27845829934 | Skype: ryan.graham.crichton
E-mail: ry...@jembi.org

--
You received this message because you are subscribed to the Google Groups "OpenHIM-Implementers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openhim-implemen...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
--
Ryan Crichton
Lead Developer, Jembi Health Systems |  SOUTH AFRICA
Mobile: +27845829934 | Skype: ryan.graham.crichton
E-mail: ry...@jembi.org

--
You received this message because you are subscribed to the Google Groups "OpenHIM-Implementers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openhim-implemen...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Ally Shaban

unread,
Sep 27, 2017, 7:52:27 AM9/27/17
to Ryan Crichton, OpenHIM-Implementers
The server has a total memory of 2GB. I have 7 channels pointing to one mediator, the channel that has a highest number of orchestration requests is the one that performs 100 orchestration requests for every single facility,i currently have 321 facilities and this is expected to increase as systems scale to other facilities,we do expect to go up to atleast 8700 facilities in the future. These channels are running at different times.

Thank you,

Ally

To unsubscribe from this group and stop receiving emails from it, send an email to openhim-implementers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OpenHIM-Implementers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openhim-implementers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
--
Ryan Crichton
Lead Developer, Jembi Health Systems |  SOUTH AFRICA
Mobile: +27845829934 | Skype: ryan.graham.crichton
E-mail: ry...@jembi.org

--
You received this message because you are subscribed to the Google Groups "OpenHIM-Implementers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openhim-implementers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
--
Ryan Crichton
Lead Developer, Jembi Health Systems |  SOUTH AFRICA
Mobile: +27845829934 | Skype: ryan.graham.crichton
E-mail: ry...@jembi.org

--
You received this message because you are subscribed to the Google Groups "OpenHIM-Implementers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openhim-implementers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages