Two small improvements

[heisee]

Hi Alex,

I just found another small subtle bug, that took me quite some time to
find out...

Line #3 in juggernaut.rb
require "json"
is unnecessary (at least with Rails 2.x).

In fact, it even breaks the definitions made for the json serializer
that were added in ActiveSupport.
Simply try this
>> script/runner "{}.to_json({})"
with and without the
require "json"
line.
Without it it'll work, but when it's there, it'll say " wrong argument
type Hash (expected Data) (TypeError)".

I'm not sure, but I think this line is required for Rails 1.x.

Another suggestion I'd like to see is is having ERB in YAML for the
juggernaut_hosts.yml file:
So instead of
CONFIG = YAML::load_file("#{RAILS_ROOT}/config/
juggernaut_hosts.yml").freeze
it could help in some cases if it was
CONFIG = YAML::load(ERB.new(IO.read("#{RAILS_ROOT}/config/
juggernaut_hosts.yml")).result).freeze

(on line #7 in juggernaut.rb)

thanks, Heiko Seebach

[heisee]

Hi again,

I also notived, that the logger is initialized again and again, even
if the log file is written correctly.
Simply put e.g.
puts "init logger again"
behind the line
return @@logger if defined?(@@logger)
in juggernaut.rb

I've no idea why, but Iit seems like
defined?(@@logger)
returns nil, even if @@logger is defined one iteration before.
Does anybody else also see this phenomenon?

Of course, this slows down juggernaut a lot...

thanks, Heiko

[heisee]

I made some performance and stress tests with the juggernaut server,
and was unable to connect with more than 1024 clients . In the EM docs
there's the solution: use epoll() instead of select() - but it
requires a 2.6 kernel.
http://www.rubyeventmachine.com/browser/trunk/EPOLL

Here's a patch, that makes EM use epoll() instead of select().

Depending on the current Linux distribution and configuration, you
might have to edit /etc/security/limits.conf
and one lines above the "# End of File" add the following two lines
* hard nofile 65535
* soft nofile 65535
(including the stars!)
On some distributions (e.g. Ubuntu) you also have to add the following
line
session required pam_limits.so
to the file /etc/pam.d/common-session .
Then reboot...
This step is only necessary, if the output of "New descriptor-table
size is..." is still lower than what was defined as parameter.


Index: lib/juggernaut/runner.rb
===================================================================
--- lib/juggernaut/runner.rb (revision 131)
+++ lib/juggernaut/runner.rb (working copy)
@@ -62,10 +62,16 @@
stop
exit
}
-
+
+ if options[:descriptor_table_size]
+ EM.epoll
+ new_size =
EM.set_descriptor_table_size( options[:descriptor_table_size] )
+ logger.debug "New descriptor-table size is #{new_size}"
+ end
EventMachine::run {

EventMachine::add_periodic_timer( options[:cleanup_timer].to_i )
{ Juggernaut::Client.send_logouts_after_timeout }
EventMachine::start_server(options[:host],
options[:port].to_i, Juggernaut::Server)
+ EM.set_effective_user( options[:user] ) if options[:user]
}
end

@@ -105,6 +111,10 @@
opts.on("-p", "--port PORT", Integer, "Specify port",
"(default: #{options[:port]})") do |v|
options[:port] = v
end
+
+ opts.on("-s", "--fdsize FILE_DESCRIPTOR_SIZE", Integer, "Set
the file descriptor size (e.g. 60000) and use epoll() on a 2.6 Linux
kernel ", "(default: use select() which is limited to 1024 clients")
do |v|
+ options[:descriptor_table_size] = v
+ end

opts.separator ""; opts.separator "Daemonization:"

Alex, maybe you could merge this into the trunk?

thanks, Heiko

[Alex MacCaw]

Ok, I'll commit these to trunk.
Thanks!

Alex

--
http://www.eribium.org | http://juggernaut.rubyforge.org | http://www.aireohq.com | Skype: oldmanorhouse

[Srikanth Pagadala]

 

Hi Alex

 

Is this available in the trunk now?

 

Hi Heisee

 

Can you please share with us how you conducted load test?

Couple of months ago, i tried to perform similar load test (see this: http://groups.google.com/group/Juggernaut-for-Rails/msg/e573a59e53413b29 ). After this i tried to device many innovative ways to load test JUG in vain. Nothing worked in a assertive way.

 

If you could share your methodology, i could continue what you have done and publish new results.

 

Ultimately, if new results are not good enough to give smoke to other push implementations out there, then may be we can help Alex build somekind of clustering Jug configuration, to turboboost Jug ....

 

thanks

[koko]

Hi Heisee,
I join Srikanth's call - load test code could be great,
infact, making it a part of the trunk makes sense...

dor

On Apr 12, 7:29 am, "Srikanth Pagadala" <srikanth.pagad...@gmail.com>
wrote:

> Hi Alex
>
> Is this available in the trunk now?
>
> Hi Heisee
>
> Can you please share with us how you conducted load test?
> Couple of months ago, i tried to perform similar load test (see this:http://groups.google.com/group/Juggernaut-for-Rails/msg/e573a59e53413b29).
> After this i tried to device many innovative ways to load test JUG in vain.
> Nothing worked in a assertive way.
>
> If you could share your methodology, i could continue what you have done and
> publish new results.
>
> Ultimately, if new results are not good enough to give smoke to other push
> implementations out there, then may be we can help Alex build somekind of
> clustering Jug configuration, to turboboost Jug ....
>
> thanks
>

> On Wed, Apr 9, 2008 at 3:13 PM, Alex MacCaw <macc...@gmail.com> wrote:
> > Ok, I'll commit these to trunk.
> > Thanks!
>
> > Alex
>

> > On Tue, Apr 8, 2008 at 11:29 PM, heisee <heiko.seeb...@googlemail.com>

[Alex MacCaw]

I committed it to the trunk last night.

Also, here's the load testing code that Heiko wrote:

require "socket"

class JugClient

 def run

   puts "start"

   threads = []

   1000.times do |a|

    threads << Thread.new(a) {            socket=TCPSocket.new("localhost",5000)

       puts "start of thread #{a}"

       s="""{

         \"command\": \"subscribe\",

         \"session_id\": \"abcef\",

         \"client_id\": #{a},

         \"channels\": [\"any\"]

         }\0

       """.gsub(/\n/, ' ')

       b=socket.send(s,s.length)

       puts "connecting client nr. #{a} and wrote #{b} bytes"

       socket.flush

       while true do

         s=socket.recv 1000

         puts "client nr. #{a} says #{s}" unless s.empty?

         return if s =~ /close/

       end

       socket.close

    }

   end

   threads.each {|t| t.join }

   puts "end"

 end

end

JugClient.new.run

[heisee]

On Apr 12, 6:29 am, "Srikanth Pagadala" <srikanth.pagad...@gmail.com>
wrote:
> Hi Heisee

>
> If you could share your methodology, i could continue what you have done and
> publish new results.

Hi Srikanth,
in addition to the test program I that's listed here, I'd like to give
you some more information about what I found out yet:

The problem with it (at least on my machine) is, that when it
connected around a several hundred clients, it stopped working.
For me it looks like if eventmachine won't call the eventhandlers
correct (after a hundred times where it behaves correctly):
"post_init" get's called always (the client prints out "connecting
client nr. 1000 and wrote ... bytes", but the subscribe command isn't
called for all clients, because the event handelt "receive_data" isn't
called any more - even if the bytes are written by the client)
But sometimes it's also running fine with 1000 clients.
Maybe the test program behaves different than flash, I'm not sure
about it. But according to the messages, that the server prints out,
it behaves the same, except the initial policy request that I left
out...

I'm very interested if you (or any of you readers) can see similar
results on your machines. I didn't test it with the select method, I
always used the new epoll mechanism - see my other post above. Alex is
about putting the patch into the trunk...

Have fun,
Heiko Seebach

[koko]

hi there,
i've been spending the last day trying to make the same setting on an
open-solaris machine (joyent accelerator) but without any luck...
I finally tried "prctl -n process.max-file-descriptor -r -v 4096 $$"
but even when it was set to 1024, it kept dying on me on ~255

do you have any experience/knowledge/idea what can be done?

10x!

On Apr 9, 1:29 am, heisee <heiko.seeb...@googlemail.com> wrote:
> I made some performance and stress tests with the juggernaut server,
> and was unable to connect with more than 1024 clients . In the EM docs

> there's the solution: useepoll() instead of select() - but it
> requires a 2.6 kernel.http://www.rubyeventmachine.com/browser/trunk/EPOLL
>
> Here's a patch, that makes EM useepoll() instead of select().

> the file descriptor size (e.g. 60000) and useepoll() on a 2.6 Linux

[koko]

Ok,
some resolutions:

1. the load testing above will never go beyond 255~ because of ruby
limits,
so to commit real test set the 1000.times line to 200.times and run it
in multiple instances until it can't get any more requests
2. epoll is called automatically on machines that allows it (look at
the eventmachine source code) so no need to specifically call it a.k.a
modify the code
3. still investigating ;-)

enjoy the passover!
koko

[koko]

(that doesn't mean that options[:descriptor_table_size] shouldn't be
set, it should)

[heisee]

Hi koko,

> 1. the load testing above will never go beyond 255~ because of ruby
> limits,

ah, that's a good point.
I repeated the testes with setting the loop to 200.times and started
the jug_client 10 times.
I saw 2 different effects (both after a little more than 1024
connections):
Sometimes, the juggernaut server just terminated without anylog
output, and sometimes it simply hang. An strace showed
gettimeofday({1208953671, 576842}, NULL) = 0
select(1026, [4], [], [], {0, 12054}) = 1 (left {0, 12054})
gettimeofday({1208953671, 577030}, NULL) = 0
select(1026, [4], [], [], {0, 11867}) = 1 (left {0, 11867})
gettimeofday({1208953671, 577219}, NULL) = 0
select(1026, [4], [], [], {0, 11677}) = 1 (left {0, 11677})
gettimeofday({1208953671, 577407}, NULL) = 0
...
and so on.
I've ne good idea on how to track that. Anybody else?

thanks, Heiko

[Srikanth Pagadala]

 

So finally, even i restarted my efforts to load test Jug today. Unfortunately i hit a wall right away ..

 

My rails/jug setup is working fine. Following lines are seen on the server logs for a request from the rails app.

 

srikanth@srikanth-laptop:~/workspace/Test$ juggernaut -c juggernaut.yml
Starting Juggernaut server on port: 5001...
D, [2008-04-24T00:23:21.129784 #11225] DEBUG -- : New client [10.21.112.9]
D, [2008-04-24T00:23:21.295111 #11225] DEBUG -- : Receiving data: <policy-file-request/>
D, [2008-04-24T00:23:21.295444 #11225] DEBUG -- : Processing message: <policy-file-request/>
D, [2008-04-24T00:23:21.295688 #11225] DEBUG -- : Sending crossdomain file
D, [2008-04-24T00:23:22.219271 #11225] DEBUG -- : New client [10.21.112.9]
D, [2008-04-24T00:23:22.420996 #11225] DEBUG -- : Receiving data: {"command": "subscribe", "session_id": "fd7329fec90aa3ffb932ebc1-215371", "channels": [2]}
D, [2008-04-24T00:23:22.421429 #11225] DEBUG -- : Processing message: {"command": "subscribe", "session_id": "fd7329fec90aa3ffb932ebc1-215371", "channels": [2]}

 

Which means everything is magnificent. :)

 

And following lines appear for a request from my java based load testing socket program:

D, [2008-04-24T00:24:21.671802 #11225] DEBUG -- : New client [10.21.112.9]
D, [2008-04-24T00:24:21.873734 #11225] DEBUG -- : Receiving data: {"session_id":"3905a9deec852ef2:-5e1ac3c9:1197f53da40:-8000","command":"subscribe","channels":[1]}

 

Alex, do you know why didn't Jug server show any "Processing message" log statement for the request from test program?

there must be something wrong in my handshaking protocol...

 

Snippet from my socket code is below:

 

this.os = new PrintStream(this.socket.getOutputStream());

 

Map map = new HashMap();
map.put("command", "subscribe");
map.put("session_id", new VMID().toString());
map.put("channels", new int[] { 1 });

String subscribeCmd = JSONSerializer.toJSON(map).toString();
this.os.write(subscribeCmd.getBytes());
this.os.flush();

[Srikanth Pagadala]

 

ha figured it out.

we need a "\0" at the end of the payload.

why do we need a "\0" in the end, again?????

[Alex MacCaw]

XmlSocket (flash) appends this automatically - I just figured it was a convenient delimiter.

[Srikanth Pagadala]

 

my preliminary tests also stumbled around 1000 concurrent connections mark.

i didnt get enough time to tune other OS prarameters etc, but it does appear like 1000 is the best we have so far.

I'll conduct more tests shortly.

 

Atleast good news is Java didnt have any limitation on number of sockets opening in a process.

 

I'll share my Java based Jug Test framework once it matures little bit.