out of memory error

[gilado]

I want to create and array of strings sorted by popularity.

I get an out of memory error, eventhough top command shows jsdb is consuming less than 100MB of RAM (out of 2GB available to the jsdb process)

Is there a way to tell jsdb to use more memory ?

The strings are stored in a database and require some pre processing

IT works for a smallish array (70,000) but runs out of memory with a larger array.

Here's simplified code

// Get the strings
var tc = db.query("SELECT mycolumn FROM mytable");

// Create a hash by string with associated count
var H = {};
for (var i=1; i<=tc.count; i++) {
    var s = tc.data(i,1);
    // Modify s
    H[s] = isNaN(H[s]) ? 1 :H[s]++;
}

// Create an Array of objects, each holding a string and a count
var A = [];
for (var s in H) A.push({str : s, cnt : H[s]}); // Here it runs out of memory

// Sort H in A by popularity: entries with highest count first
A.sort(function(a,b) { return b.cnt - a.cnt });


Any ideas, alternative solutions appreciated. I need to sort 1000000 entries









First I create a mapping

[Shanti Rao]

JavaScript's "for (var s in H)" is really inefficient, because it creates an atom array for each index, then iterates over that.

You want to use the Array.map() function instead. Or not use it at all! This should be more efficient:

H = {}
A = []

for (...) {
if (s in H)
   H[s].cnt++
else
   H[s]={str:s,cnt:1}
}

A.sort(...)

[gilado]

Thanks Shanti!  I used your idea and created a function to populate both arrays at the same time

function AddItemToTables(H,A,item)
{
    if (H[item])
        A[H[item]].cnt++;
    else {
        H[item] = A.length;
        A[A.length] = { 'item': item, 'cnt': 1 };
    }
}

Above if the item has already been inserted once, H[item] holds its location in A[].  This removes the need to do for (var i in H) and A[] can be sorted after all items have been added.

The reason for using two tables is that if item is a string representing a number then A['17']=1 and A[17]=1 both are stored as an element of the array not a property of the object.

The program has two sets of objects LX={},LXA=[] and LM={},LMA=[] and inserts elements into both.  The number of insertion per input item is not 1:1.

It runs out of memory after inserting

[00:30:31.462][028.288] 63269: LXA.length 28614 LMA.length 136129
../mklistlm.js:109      out of memory

I added printfs to the functions spider monkey uses to get memory in js/src/jsutil.h and it does not really run out of memory; some other internal resources is exhausted.

Here are the mods:


static JS_INLINE void* js_malloc(size_t bytes) {
    if (bytes < sizeof(void*)) /* for asyncFree */
        bytes = sizeof(void*);
    void *p = malloc(bytes);
printf("malloc b=%d p=%p \n",bytes,p);
    return p;
}

static JS_INLINE void* js_calloc(size_t bytes) {
    if (bytes < sizeof(void*)) /* for asyncFree */
        bytes = sizeof(void*);
    void *p = calloc(bytes, 1);
printf("calloc b=%d p=%p \n",bytes,p);
    return p;
}

static JS_INLINE void* js_realloc(void* p, size_t bytes) {
    if (bytes < sizeof(void*)) /* for asyncFree */
        bytes = sizeof(void*);
    void *n = realloc(p, bytes);
printf("realloc b=%d p=%p n=%p \n",bytes,p,n);
    return n;
}

static JS_INLINE void js_free(void* p) {
    free(p);
printf("free p=%p \n",p);
}

And the tail of the  printout

realloc b=32 p=(nil) n=0xaa35ad8
realloc b=16 p=(nil) n=0xb175e50
malloc b=16 p=0x9ef88c0
malloc b=16 p=0xa795b98
realloc b=16 p=0xa795b98 n=0xa795b98
malloc b=28 p=0x9e6b6d8
realloc b=786432 p=0xb6646008 n=0xb6646008
malloc b=44 p=0xa341000
malloc b=16 p=0x9f16610
realloc b=16 p=0x9f16610 n=0x9f16610
malloc b=28 p=0xa8820b8
malloc b=44 p=0xa30d368
malloc b=16 p=0xaa81138
[00:30:31.317][028.143] 63268: LXA.length 28613 LMA.length 136124
[00:30:31.462][028.288] 63269: LXA.length 28614 LMA.length 136129
../mklistlm.js:109      out of memory

[Michael I]

Please try changing:

Env->rt = JS_NewRuntime(8L * 1024L * 1024L);

to e.g.
Env->rt = JS_NewRuntime(64L * 1024L * 1024L);

in file jsdb.cpp

Hope it helps

Michael

[gilado]

So it turns out the amount of memory is limited to 8MB in

 JS_NewRuntime(8L * 1024L * 1024L);

In jsdb.cpp line 577.

I changed that to 200MB (200L * 1024L * 1024L) and my program completed successfully.

[gilado]

Thanks Michael!

Are there any drawbacks to increasing this value? That is, should I create a configuration setting (e.g. via an environment variable) for this value, or should I just set it to 200M)

[Michael I]

Glad I could help you !

I'm using greater value (64L) for quite some time and I didnt notice any problems (I had also problems with out of memory error)
The only problem I can think of is when the computer has less memory available.
Firefox is using 64L (AFAIK) so I think it is safe value.
Michael