Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
Agent send-off: ~20k/s
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  11 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
MarkSwanson  
View profile  
 More options Oct 4 2009, 10:45 pm
From: MarkSwanson <mark.swanson...@gmail.com>
Date: Sun, 4 Oct 2009 19:45:34 -0700 (PDT)
Local: Sun, Oct 4 2009 10:45 pm
Subject: Agent send-off: ~20k/s
I recently integrated Clojure with two async messaging systems.
I wound up doing "send" operations through a Clojure agent.
I was curious how many agents I could spawn per second and found I
could spawn about 20K agents / second.

Code for testing:

(def t (agent true))

(defn tt [_ num]
     (try
       (let [x (inc num)])
       (catch Throwable t (println "error"))))

(defn agent-speed-test []
     (time (loop [b 1]
             (send-off t tt b)
             (if (> b 20000)
               b
               (recur (inc b))))))

After a few iterations in the repl (started with -server) my best time
was:
"Elapsed time: 1021.700789 msecs" - which is pretty much exactly 20k/
s.

CPU: 2.4GHz Core2 Duo.
Java 1.6.0_14-ea-b03
Clojure from git/master around Sept/2009.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
ngocdaothanh  
View profile  
 More options Oct 5 2009, 2:45 am
From: ngocdaothanh <ngocdaoth...@gmail.com>
Date: Sun, 4 Oct 2009 23:45:28 -0700 (PDT)
Local: Mon, Oct 5 2009 2:45 am
Subject: Re: Agent send-off: ~20k/s
I think it is not "spawn about 20K agents / second", it is 20K message
passings / second. The number is about that of Erlang.

On Oct 5, 11:45 am, MarkSwanson <mark.swanson...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
MarkSwanson  
View profile  
 More options Oct 5 2009, 11:51 am
From: MarkSwanson <mark.swanson...@gmail.com>
Date: Mon, 5 Oct 2009 08:51:19 -0700 (PDT)
Local: Mon, Oct 5 2009 11:51 am
Subject: Re: Agent send-off: ~20k/s
On Oct 5, 2:45 am, ngocdaothanh <ngocdaoth...@gmail.com> wrote:

> I think it is not "spawn about 20K agents / second", it is 20K message
> passings / second. The number is about that of Erlang.

As Clojure uses a thread pool for agents I agree 'spawn' was the wrong
word. Thanks for the correction.

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
John Harrop  
View profile  
 More options Oct 5 2009, 3:18 pm
From: John Harrop <jharrop...@gmail.com>
Date: Mon, 5 Oct 2009 15:18:34 -0400
Local: Mon, Oct 5 2009 3:18 pm
Subject: Re: Agent send-off: ~20k/s

On Mon, Oct 5, 2009 at 11:51 AM, MarkSwanson <mark.swanson...@gmail.com>wrote:

> On Oct 5, 2:45 am, ngocdaothanh <ngocdaoth...@gmail.com> wrote:
> > I think it is not "spawn about 20K agents / second", it is 20K message
> > passings / second. The number is about that of Erlang.

> As Clojure uses a thread pool for agents I agree 'spawn' was the wrong
> word. Thanks for the correction.

Some confusion here may also be from the subject line. It mentions send-off,
which actually can spawn unlimited numbers of new threads. Regular send
doesn't, using a fixed thread pool instead.

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
MarkSwanson  
View profile  
 More options Oct 6 2009, 10:35 am
From: MarkSwanson <mark.swanson...@gmail.com>
Date: Tue, 6 Oct 2009 07:35:21 -0700 (PDT)
Local: Tues, Oct 6 2009 10:35 am
Subject: Re: Agent send-off: ~20k/s
Thanks John.
I was curious about the details so I took a dive in to the source to
see for myself.
In case anyone else stumbles upon this here's what I found:

In Agent.java, the number of worker threads for (send) are defined
like this:

final public static ExecutorService pooledExecutor =
        Executors.newFixedThreadPool(2 + Runtime.getRuntime
().availableProcessors());

The clojure (send) calls Java Agent dispatch(), which winds up using
the pooledExecutor.

Clojure (send-off) follows the same path but winds up using the
soloExecutor - which can spawn (and temporarily cache) an unlimited
number of threads as required:

http://java.sun.com/javase/6/docs/api/java/util/concurrent/Executors....()

Random thought: Let's test (send) vs (send-off). New results using
(send):
(the -server jvm produces some wild results for a bit then I get
something crazy: 7.9ms):

Clojure=> (agent-speed-test)
"Elapsed time: 1164.702387 msecs"
20001
Clojure=> (agent-speed-test)
"Elapsed time: 1416.682837 msecs"
20001
Clojure=> (agent-speed-test)
"Elapsed time: 7.907515 msecs"   <------ ** ???? LOL **

This happens infrequently, but it did happen more than once.

Also, I note that only 2 cores are used on my quad core2 duo. Weird.
It's as if
 Runtime.getRuntime().availableProcessors()  fails (returns zero?)


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Christophe Grand  
View profile  
 More options Oct 6 2009, 11:18 am
From: Christophe Grand <christo...@cgrand.net>
Date: Tue, 6 Oct 2009 17:18:33 +0200
Local: Tues, Oct 6 2009 11:18 am
Subject: Re: Agent send-off: ~20k/s

Hi,

Since you are sending all actions to the same agent they are enqueued and
processed sequentially. That's why you are seeing only two active cores (one
for the main thread (repl) or GC and one for the agent).

On Tue, Oct 6, 2009 at 4:35 PM, MarkSwanson <mark.swanson...@gmail.com>wrote:

--
Professional: http://cgrand.net/ (fr)
On Clojure: http://clj-me.blogspot.com/ (en)

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
MarkSwanson  
View profile  
 More options Oct 6 2009, 1:48 pm
From: MarkSwanson <mark.swanson...@gmail.com>
Date: Tue, 6 Oct 2009 10:48:53 -0700 (PDT)
Local: Tues, Oct 6 2009 1:48 pm
Subject: Re: Agent send-off: ~20k/s
Good catch. I had forgotten that.

I did some more tests with 4 queues and found that there seemed to be
some contention going on that prevented all cores from being utilized
fully.
With the example provided 2 cores will pin at 100% (excellent). With 4
atoms and one test fn one core will stay around 20%, no other core >
70%.
When I used 4 separate worker fns along with 4 separate atoms things
improved and all 4 cores pinned at around 60%. It would be interesting
to know why having 4 separate fns makes a difference (STM?).

Wrt 60%: I simply think this was an artifact of my test. If I had
spawned another thread to feed another atom I'm sure I could have
easily pegged all CPUs to 100%.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Christophe Grand  
View profile  
 More options Oct 6 2009, 2:37 pm
From: Christophe Grand <christo...@cgrand.net>
Date: Tue, 6 Oct 2009 20:37:18 +0200
Local: Tues, Oct 6 2009 2:37 pm
Subject: Re: Agent send-off: ~20k/s

Your worker fn is too quickly executed: more time is spent managing queues
(and that require some synchronization) than executing actions.
If your worker fn was more realistic (more computationally heavy) I tjink
you'll see 4 cores humming at 100%.

(You wrote "atom" several times but I guess you meant "agent".)

On Tue, Oct 6, 2009 at 7:48 PM, MarkSwanson <mark.swanson...@gmail.com>wrote:

--
Professional: http://cgrand.net/ (fr)
On Clojure: http://clj-me.blogspot.com/ (en)

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
MarkSwanson  
View profile  
 More options Oct 6 2009, 11:58 pm
From: MarkSwanson <mark.swanson...@gmail.com>
Date: Tue, 6 Oct 2009 20:58:18 -0700 (PDT)
Local: Tues, Oct 6 2009 11:58 pm
Subject: Re: Agent send-off: ~20k/s
> (You wrote "atom" several times but I guess you meant "agent".)

Heh heh... yeah.

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
ngocdaothanh  
View profile  
 More options Oct 7 2009, 2:59 am
From: ngocdaothanh <ngocdaoth...@gmail.com>
Date: Tue, 6 Oct 2009 23:59:22 -0700 (PDT)
Local: Wed, Oct 7 2009 2:59 am
Subject: Re: Agent send-off: ~20k/s
Mark,
What tool did you use to see the CPU utilization, especially when the
test run time is so short?

Just curious.

On Oct 7, 12:58 pm, MarkSwanson <mark.swanson...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
MarkSwanson  
View profile  
 More options Oct 7 2009, 10:17 am
From: MarkSwanson <mark.swanson...@gmail.com>
Date: Wed, 7 Oct 2009 07:17:04 -0700 (PDT)
Local: Wed, Oct 7 2009 10:17 am
Subject: Re: Agent send-off: ~20k/s

On Oct 7, 2:59 am, ngocdaothanh <ngocdaoth...@gmail.com> wrote:

> Mark,
> What tool did you use to see the CPU utilization, especially when the
> test run time is so short?

> Just curious.

I used the CPU utilization monitor in Linux.
I adjusted the loop so it took about 5-7 seconds.

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »