APE 1.1.0

[Doug]

I've been experimenting with the new APE 1.1.0, and it appears that it
works until memory consumption of the aped process reaches 1.9% and
then whenever I attempt to post some data I get

Array ( [time] => 1301590992 [raw] => ERR [data] => Array ( [code] =>
001 [value] => BAD_PARAMS ) ) )

I can open a channel and I see this in ape.log:

2011-03-30 15:58:21 - src/users.c:180 - New user - (ip : 2.my.ip.69)

So it appears to recognize that somebody has joined the channel and is
listening. Also, the page I have set up that listens to the channel
sends no errors, it includes the js file and its allocated a numbered
subdomain like it should (4.mydomain.com, for example).

My main.ape.js looks like this:

Ape.addEvent("init", function() {
include("framework/mootools.js");
include("framework/Http.js");
//include("framework/userslist.js");
include("utils/utils.js");
include("commands/proxy.js");
include("commands/inlinepush.js");
//include("examples/nickname.js");
//include("examples/move.js");
//include("utils/checkTool.js"); //Just needed for the APE JSF
diagnostic tool, once APE is installed you can remove it
//include("examples/ircserver.js");
//include("framework/http_auth.js");
});

I've commented out some stuff that I didn't think I needed. This
configuration works on an earlier installation (1.01a I believe). The
problem with the earlier version is memory consumption, but it seems
to happen at a higher percentage (round about 15-20%).

I have tried using the older JSF and the newest one I could find on
github, behavior appears to be identical.

I have fixed the older installation by running a cronjob every 6 hours
to restart the ape process.

Any advice or suggestions would be a huge help, thank you!

[Anthony Catel]

Hi doog,

You don't have any custom server-side javascript?

How many concurent users do you have?
What is your CPU architecture?

Thanks

Le 31/03/2011 19:18, Doug a �crit :

[Doug]

Hi Anthony,

Thanks for the response, I am testing with 4 concurrent users.

I am working with a Fedora, Amazon Linux AMI, EC2 instance, here's
some specs:

Small Instance – default*
1.7 GB memory
1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit)
160 GB instance storage
32-bit platform
I/O Performance: Moderate
API name: m1.small

I have added a function that wraps the APE.Client(); code (I only run
startRT() from the listener page):

function startRT(channel, callback) {
var client = new APE.Client();
client.load();
client.addEvent('load', function() {
client.core.start();
});
client.addEvent('ready', function() {
myJoin(client , channel);
client.onRaw('data', function(raw, pipe) {
callback(raw.data);
});
});
return client;
}

function myJoin(client, channel) {

if(isArray(channel) == false) {
channel = removeNonAlphas(channel);
client.core.join(channel);
}
else {
for (var i in channel) {
channel[i] = removeNonAlphas(channel[i]);
}
client.core.join(channel);
}
var channelpipe = channel.toLowerCase();
}

function removeNonAlphas(channel){
channel = channel.replace(/[^a-zA-Z 0-9]+/g,'');
return channel;
}

function isArray(obj) {
if (obj.constructor.toString().indexOf("Array") == -1)
return false;
else
return true;
}

Thank you!

[Anthony Catel]

So, If I understand correctly, at the begening everything works fine and
sudently stop working with a "BAD_PARAMS".
You don't have any server-side script?

Le 31/03/2011 22:05, Doug a �crit :

> Hi Anthony,
>
> Thanks for the response, I am testing with 4 concurrent users.
>
> I am working with a Fedora, Amazon Linux AMI, EC2 instance, here's
> some specs:
>

> Small Instance � default*

[Doug]

I'm sorry I must have misunderstood, the way I am sending the events
is through a PHP script similar to the one that I found on APE's
website that does a single POST:

<?php
$c=0;
while($c <= 5) {
$APEserver = 'http://mydomain:6969/?';
$APEPassword = 'mypasswd';

$messages = array(
'Test Message',
'test2',
'test3',
);

$cmd = array(array(
'cmd' => 'inlinepush',
'params' => array(
'password' => $APEPassword,
'raw' => 'DATA',
'channel' => 'mychannelname',
'data' => array( //Note: data can't be a string
'message' => $messages[array_rand($messages)]
)
)
));
echo "<br><br>".json_encode($cmd)."<br><br>";
var_dump($APEserver.rawurlencode(json_encode($cmd)));
$data =
file_get_contents($APEserver.rawurlencode(json_encode($cmd)));
$data = json_decode($data, TRUE);

if ($data[0]->data->value == 'ok') {
echo 'Message sent!';
} else {
//echo 'Error sending message, server response is : <pre>'.
print_r($data);
echo '</pre>';
}
$c++;
}

?>

This is the page that tells me "ok" sending the message until the
server reaches 1.9% and then it returns BAD_PARAMS, the listening page
doesn't seem aware of any problems.
Thank you, regards,
Doug

On Mar 31, 2:51 pm, Anthony Catel <a.ca...@weelya.com> wrote:
> So, If I understand correctly, at the begening everything works fine and
> sudently stop working with a "BAD_PARAMS".
> You don't have any server-side script?
>

> Le 31/03/2011 22:05, Doug a crit :

>
>
>
>
>
>
>
> > Hi Anthony,
>
> > Thanks for the response, I am testing with 4 concurrent users.
>
> > I am working with a Fedora, Amazon Linux AMI, EC2 instance, here's
> > some specs:
>

> > Small Instance default*

[Doug]

Anthony,

I haven't changed any of the other APE server script, just what I got
from you in the 1.1.0 release. Perhaps I could turn some more logging
on to debug some more?

Thanks for your hard work on this project,
Regards,
Doug

[a.c...@weelya.com]

Ok, so you didn't have this issue with APE 1.0 ?
How many messages do you send before encoutering it ?

Thanks

On Fri, 1 Apr 2011 08:31:37 -0700 (PDT), Doug <wrexham...@gmail.com>
wrote:

[Doug]

Anthony,

Well a similar situation was happening with APE 1.0. APE's process
would slowly increase and eventually it reach a certain point (around
15% to 20%). At this point, I would attempt to post and it would
return "UNKNOWN CHANNEL" which would normally happen when nobody is
listening to that channel. Usually it took about 6-8 hours to reach
this point.

Under 1.1.0, it appears to be a similar time frame, (perhaps a little
longer ~10 to ~12 hours) and the memory consumption is much lower
(around 1.9%), however at this point a post-attempt will return
"BAD_PARAMS."

The test I conducted with 1.1.0 was about ten posts a second, but
under the 1.0 version the traffic was much heavier, probably around
100 a second. 1.0 seemed to handle this really well, I automatically
restarted the 1.0 server every 6 hours and it definitely did the job.

Thank you,
Doug

On Apr 1, 9:35 am, <a.ca...@weelya.com> wrote:
> Ok, so you didn't have this issue with APE 1.0 ?
> How many messages do you send before encoutering it ?
>
> Thanks
>

> On Fri, 1 Apr 2011 08:31:37 -0700 (PDT), Doug <wrexhamthe1...@gmail.com>

[a.c...@weelya.com]

Thanks ;)
I'll run the same test tonight. I give you my results tomorrow ;)

oh and BTW 32 or 64bits ?

Thanks

On Fri, 1 Apr 2011 10:21:57 -0700 (PDT), Doug <wrexham...@gmail.com>
wrote:

[Doug]

Anthony,

Excellent, sounds good, we're using a 32-bit platform.

Looking forward to hearing from you,
Doug

On Apr 1, 11:25 am, <a.ca...@weelya.com> wrote:
> Thanks ;)
> I'll run the same test tonight. I give you my results tomorrow ;)
>
> oh and BTW 32 or 64bits ?
>
> Thanks
>

> On Fri, 1 Apr 2011 10:21:57 -0700 (PDT), Doug <wrexhamthe1...@gmail.com>

[Anthony Catel]

Hi Doug,

I've been running the test during 12 hours with :

- 4 users connected
- 5 messages second (with your script running in a while(1){sleep(1) }).

It still working but take 0.4% on my 6GB memory (which is may be not
normal).

Anyway, the problem may be a garbage collection issue with the
javascript engine.
I'll do more test this night (more user, more messages) ;)

Le 01/04/2011 19:41, Doug a �crit :

[Doug]

Anthony,

Thanks so much for the tests!

These are very helpful, :)

Regards,
Doug

[Wes Garland]

On Sat, Apr 2, 2011 at 6:56 AM, Anthony Catel <a.c...@weelya.com> wrote:

Hi Doug,

I've been running the test during 12 hours with :

- 4 users connected
- 5 messages second (with your script running in a while(1){sleep(1) }).

It still working but take 0.4% on my 6GB memory (which is may be not normal).

Anyway, the problem may be a garbage collection issue with the javascript engine.
I'll do more test this night (more user, more messages) ;)

• ensure you are building --enable-threadsafe so that you get the benefit of background-thread finalization etc
  
• run a second thread in another context of the same compartment and use this to call JS_MaybeGC() on a timer (GPSEE uses a 2-second timer IIRC)
• ensure all your code in libape_spidermonkey.c which can possibly block (any I/O calls, especially into mysql, read/write sockets, files, etc) are surrounded with JS_SuspendRequest()...JS_ResumeRequest() calls
• ensure that you do not leave active contexts "floating" -- either end them, destroy them, or suspend them
  

If you follow these approaches, you will be able to do opportunistic GC, where the likelihood that you run your GC on one thread while another is in I/O increases dramatically.  So, you will run many short GC runs (hopefully!) instead of rarely running GC when you have exceeded the trigger factor.

IIRC, the default trigger factor is for the JS heap to have increased to 16 times the size it was the last time you GC'd.   You can adjust this with JS_SetGCTriggerFactor()(? -- IIRC) for JS 1.8.2, or JS_SetGCParameter(rt, JSGC_TRIGGER_FACTOR, value) for JS 1.8.5.

Do not set the trigger below 200 (meaning 200% heap growth) because a bug in the way the trigger is evaluated in jsgc.cpp will cause the same behaviour as JS_SetGCZeal(cx, 1) and kill you on perf.

You can see what's taking up your memory with JS_DumpHeap.  This works best in a debug build, and IIRC, if you have good JSClass::name, and also, if you name your GC roots.

The key to keeping your heap size down is to use as little JS memory as possible (obviously!), ensuring that you don't leave things like giant closures lying around as props of the global variable but unused.  That is for the JS coder.

From the C side, we must make sure we run the garbage collector fairly often. The opportunistic GC algorithm I described works because JSAPI uses a stop-the-world GC based on a mutex rendezvous, where the GC-initiating thread waits on the mutex until all other active contexts have reported that they are waiting for that thread to mark/sweep the roots. This is why we call JS_MaybeGC() frequently and suspend around system calls -- the other thread waiting will mark/sweep as soon as the busy context suspends, then hopefully finishes before the system call so the GC was zero-cost.  Then finalization (free) occurs in yet another thread.

Wes

--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102

[Doug]

Thanks Wes,

Were you able to reproduce the problem, Anthony?

Perhaps your higher memory (6GB) could solve / help the problem?

Thanks,
Doug

On Apr 4, 11:09 am, Wes Garland <w...@page.ca> wrote:

> On Sat, Apr 2, 2011 at 6:56 AM, Anthony Catel <a.ca...@weelya.com> wrote:
> > Hi Doug,
>
> > I've been running the test during 12 hours with :
>
> > - 4 users connected
> > - 5 messages second (with your script running in a while(1){sleep(1) }).
>
> > It still working but take 0.4% on my 6GB memory (which is may be not
> > normal).
>
> > Anyway, the problem may be a garbage collection issue with the javascript
> > engine.
> > I'll do more test this night (more user, more messages) ;)
>

>    - ensure you are building --enable-threadsafe so that you get the benefit
>    of background-thread finalization etc
>    - run a second thread in another context of the same compartment and use

>    this to call JS_MaybeGC() on a timer (GPSEE uses a 2-second timer IIRC)

>    - ensure all your code in libape_spidermonkey.c which can possibly block

>    (any I/O calls, especially into mysql, read/write sockets, files, etc) are
>    surrounded with JS_SuspendRequest()...JS_ResumeRequest() calls

>    - ensure that you do not leave active contexts "floating" -- either end

[a.c...@weelya.com]

Sorry I didn't find the time to reproduce yet.
I run the bench tonight !

On Wed, 6 Apr 2011 15:04:48 -0700 (PDT), Doug <wrexham...@gmail.com>
wrote:

[Wes Garland]

Doug, are you running any custom server-side code?

It is really easy introduce memory leaks if you are not careful.

Wes

--
You received this message because you are subscribed to the Google
Groups "APE Project" group.
To post to this group, send email to ape-p...@googlegroups.com
To unsubscribe from this group, send email to
ape-project...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/ape-project?hl=en
---
APE Project (Ajax Push Engine)
Official website : http://www.ape-project.org/
Git Hub : http://github.com/APE-Project/

[Anthony Catel]

Not yet.
I didn't find the time to run a new test. I'll do it tonight

Le 07/04/2011 00:04, Doug a �crit :

[Anthony Catel]

I managed to reproduce the bug !
I let you know when fixed

Le 07/04/2011 00:04, Doug a �crit :

[Mantux]

i think i have same problem, "Server 1.01dev"
ape inline push stops working over the time even if iddle .
here is my article:
https://groups.google.com/group/ape-project/browse_thread/thread/78fc8efba2439918?hl

say if i can help testing, im verry interested to find solution for
this,
APE is verry nice Programm, lets make it better together.

greetz Mantux.

[a.c...@weelya.com]

Hey,

I found a bug today, that *should* fix this issue.
Please apply this patch :
https://github.com/APE-Project/APE_Server/commit/31dc70af88437551b198c0b8eebe915856f32222

And let me know ;)

Thanks

Anthony Catel

[Mantux]

updatet ape from git,
now activated on the system,
about 1000 inline pushes in a minute.
and about 15 all work day active - real people, listenig clients.

[Anthony Catel]

Perfect. I also have something like that running since 24 hours. No
"BAD_PARAM" for the moment ;)

Le 13/04/2011 13:20, Mantux a �crit :

[Mantux]

running without errors or crashes since 2011-04-13 12:18:15
sockets pushing full power now about 500 connections minute, at day
time, and iddle for 4 hours at night.
ape logs have no errors, and "top |grep aped > checkape.log" newer
gone over 0,1% cpu/ram Load

NICE Job Anthony :D

[Wes Garland]

> NICE Job Anthony :D

And Wes! I think this is the rooting issue I helped troubleshoot :)

Glad to hear this is working reliably for you.  A stable server is very important for us!

Wes

--
You received this message because you are subscribed to the Google
Groups "APE Project" group.
To post to this group, send email to ape-p...@googlegroups.com
To unsubscribe from this group, send email to
ape-project...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/ape-project?hl=en
---
APE Project (Ajax Push Engine)
Official website : http://www.ape-project.org/
Git Hub : http://github.com/APE-Project/

[Anthony Catel]

Indeed. Thanks Wes for sorting this bug out !

[Pablo]

Can anyone help me with my upgrade? I'm still stuck on APE 1.0

Can't compile the spidermonkey module from git
https://groups.google.com/forum/m/?source=mog&hl=en&gl=us#!topic/ape-project/6AVLdAjfuAY

[Doug]

Excellent! Thanks for all your help Anthony, it's been running for
about 4 days now, and no BAD_PARAMS error, I've also been opening
random channels and listening. Memory consumption is at about 4% right
now, the best performance out of any APE installation though.

Thanks again!

On Apr 14, 8:39 am, Anthony Catel <a.ca...@weelya.com> wrote:
> Indeed. Thanks Wes for sorting this bug out !
>

> Le 14/04/2011 16:25, Wes Garland a �crit :

>
>
>
>
>
>
>
> > > NICE Job Anthony :D
>
> > And Wes! I think this is the rooting issue I helped troubleshoot :)
>
> > Glad to hear this is working reliably for you.  A stable server is
> > very important for us!
>
> > Wes
>

> > On Thu, Apr 14, 2011 at 9:38 AM, Mantux <mantasm...@googlemail.com
> > <mailto:mantasm...@googlemail.com>> wrote:
>
> >     running without errors or crashes since 2011-04-13 12:18:15
> >     sockets pushing full power now about 500 connections minute, at day
> >     time, and iddle for 4 hours at night.
> >     ape logs have no errors, and "top |grep aped > checkape.log" newer
> >     gone over 0,1% cpu/ram Load
>
> >     NICE Job Anthony :D
>
> >     On 13 Apr., 18:19, Anthony Catel <a.ca...@weelya.com

> >     <mailto:a.ca...@weelya.com>> wrote:
> >     > Perfect. I also have something like that running since 24 hours. No
> >     > "BAD_PARAM" for the moment ;)
>
> >     > Le 13/04/2011 13:20, Mantux a crit :
>
> >     > > updatet ape from git,
> >     > > now activated on the system,
> >     > > about 1000 inline pushes in a minute.
> >     > > and about 15 all work day active - real people, listenig clients.
>
> >     --
> >     You received this message because you are subscribed to the Google
> >     Groups "APE Project" group.
> >     To post to this group, send email to ape-p...@googlegroups.com

> >     <mailto:ape-p...@googlegroups.com>

> >     To unsubscribe from this group, send email to
> >     ape-project...@googlegroups.com

> >     <mailto:ape-project%2Bunsu...@googlegroups.com>

> >     For more options, visit this group at
> >    http://groups.google.com/group/ape-project?hl=en
> >     ---
> >     APE Project (Ajax Push Engine)
> >     Official website :http://www.ape-project.org/
> >     Git Hub :http://github.com/APE-Project/
>
> > --
> > Wesley W. Garland
> > Director, Product Development
> > PageMail, Inc.

> >+1 613 542 2787x 102

[Mantux]

STILL RUNNING STABLE FROM 2011-04-13 12:18:15

thx Anthony and Wes, now ape runs like a charm, :D
mail me if need some testing

On 14 Apr., 16:39, Anthony Catel <a.ca...@weelya.com> wrote:
> Indeed. Thanks Wes for sorting this bug out !
>

> Le 14/04/2011 16:25, Wes Garland a �crit :

>
> > > NICE Job Anthony :D
>
> > And Wes! I think this is the rooting issue I helped troubleshoot :)
>
> > Glad to hear this is working reliably for you.  A stable server is
> > very important for us!
>
> > Wes
>

> > On Thu, Apr 14, 2011 at 9:38 AM, Mantux <mantasm...@googlemail.com
> > <mailto:mantasm...@googlemail.com>> wrote:
>
> >     running without errors or crashes since 2011-04-13 12:18:15
> >     sockets pushing full power now about 500 connections minute, at day
> >     time, and iddle for 4 hours at night.
> >     ape logs have no errors, and "top |grep aped > checkape.log" newer
> >     gone over 0,1% cpu/ram Load
>
> >     NICE Job Anthony :D
>
> >     On 13 Apr., 18:19, Anthony Catel <a.ca...@weelya.com

> >     <mailto:a.ca...@weelya.com>> wrote:
> >     > Perfect. I also have something like that running since 24 hours. No
> >     > "BAD_PARAM" for the moment ;)
>
> >     > Le 13/04/2011 13:20, Mantux a crit :
>
> >     > > updatet ape from git,
> >     > > now activated on the system,
> >     > > about 1000 inline pushes in a minute.
> >     > > and about 15 all work day active - real people, listenig clients.
>
> >     --
> >     You received this message because you are subscribed to the Google
> >     Groups "APE Project" group.
> >     To post to this group, send email to ape-p...@googlegroups.com

> >     <mailto:ape-p...@googlegroups.com>

> >     To unsubscribe from this group, send email to
> >     ape-project...@googlegroups.com

> >     <mailto:ape-project%2Bunsu...@googlegroups.com>