Received: by 10.204.10.86 with SMTP id o22mr24122bko.18.1306377713554;
        Wed, 25 May 2011 19:41:53 -0700 (PDT)
X-BeenThere: kafka-dev@googlegroups.com
Received: by 10.204.38.88 with SMTP id a24ls391781bke.1.gmail; Wed, 25 May
 2011 19:41:52 -0700 (PDT)
Received: by 10.204.172.130 with SMTP id l2mr19932bkz.23.1306377712714;
        Wed, 25 May 2011 19:41:52 -0700 (PDT)
Received: by 10.204.172.130 with SMTP id l2mr19931bkz.23.1306377712658;
        Wed, 25 May 2011 19:41:52 -0700 (PDT)
Return-Path: <neha.narkh...@gmail.com>
Received: from mail-fx0-f50.google.com (mail-fx0-f50.google.com [209.85.161.50])
        by gmr-mx.google.com with ESMTPS id l1si97757bkl.1.2011.05.25.19.41.52
        (version=TLSv1/SSLv3 cipher=OTHER);
        Wed, 25 May 2011 19:41:52 -0700 (PDT)
Received-SPF: pass (google.com: domain of neha.narkh...@gmail.com designates 209.85.161.50 as permitted sender) client-ip=209.85.161.50;
Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of neha.narkh...@gmail.com designates 209.85.161.50 as permitted sender) smtp.mail=neha.narkh...@gmail.com; dkim=pass (test mode) header...@gmail.com
Received: by mail-fx0-f50.google.com with SMTP id 16so492576fxm.37
        for <kafka-dev@googlegroups.com>; Wed, 25 May 2011 19:41:52 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=zvyto/J3HSI3u5h9/OM/b4UKV4xo7R/fjySfwfGmbXk=;
        b=eJom+9uap+37xeM37RI6wgKhVcEURZP5VxLpljgGQf6KL1D29eakMPMm4/87/wjoAg
         bxBvCLJ3EVCtgxvXeXk5WjLiDsg/IKkQFTT5WhRf1yO8qILza8CkeybFk0/9O4n6iNjI
         pKOHeIavxhFVb/lA5amt+tl3t/cn5/3ACNZ4E=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=c2ps4DYffFlfali1Sr2tW8Nn9/QDMJvFx7aYznoUdEP2w2N1rHu1R404huHBVUJWp8
         HnEBL3hAVLYbgIzqHBS5sey4Owiuyrg6+d9MVxejTU1zwtxoFe0e4NtUe4lttiECuFF8
         z7xZ9vvg1H9G8t31oFShKp5e7V3MaWnrBM6/U=
MIME-Version: 1.0
Received: by 10.223.92.154 with SMTP id r26mr364046fam.35.1306377712405; Wed,
 25 May 2011 19:41:52 -0700 (PDT)
Received: by 10.223.151.2 with HTTP; Wed, 25 May 2011 19:41:52 -0700 (PDT)
In-Reply-To: <e4afb206-8ce2-4f57-bb7e-a2d97c3ba...@r27g2000prr.googlegroups.com>
References: <aaa854e6-b85a-4f43-ab40-6c8892d3a...@q12g2000prb.googlegroups.com>
	<BANLkTinVE+rmHWUV00qxScYBszUAEAR...@mail.gmail.com>
	<48fb8678-4719-4f11-b302-30080eb6f...@q12g2000prb.googlegroups.com>
	<e4afb206-8ce2-4f57-bb7e-a2d97c3ba...@r27g2000prr.googlegroups.com>
Date: Wed, 25 May 2011 19:41:52 -0700
Message-ID: <BANLkTimcUu5sFQRVGxZo0FFwOQUdVVj...@mail.gmail.com>
Subject: Re: [kafka-dev] Re: Slow throughput on execute kafka-producer-perf-test.sh
From: Neha Narkhede <neha.narkh...@gmail.com>
To: kafka-dev@googlegroups.com
Content-Type: multipart/alternative; boundary=00151747ba967d8b6804a424c3bf

--00151747ba967d8b6804a424c3bf
Content-Type: text/plain; charset=ISO-8859-1

To simulate the batching in run-simulator.sh, the command to use would be -

./bin/kafka-producer-perf-test.sh --brokerinfo broker.list=0:localhost:9092
--messages 100000 --topic test --threads 1 --message-size 200 --async
--batch-size 200 --delay-btw-batch-ms 6

Also, I do see what you mentioned. The producer throughput numbers,(even
with log.flush.interval=1000) is much lower.

Total Num Messages: 100000 bytes: 10000000 in 5.691 secs
Messages/sec: 17571.6043
MB/sec: 1.6758

And, for sending message one by one using this command -

./bin/kafka-producer-perf-test.sh --brokerinfo broker.list=0:localhost:9092
--messages 100000 --topic test --threads 1 --message-size 100

The numbers are not much different, just a little lower -

Messages/sec: 15181.4179
MB/sec: 1.4478

There seems to be something wrong going on inside the Producer, and this
will need more investigation.

Thanks,
Neha

On Wed, May 25, 2011 at 7:09 PM, lurga <lurga....@gmail.com> wrote:

> I see, kafka.perf.producer.Producer has a prameter
> "batchSize" (default value = 200). If I send message one by one, I
> have to change log.flush.interval.

--00151747ba967d8b6804a424c3bf
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

To simulate the batching in run-simulator.sh, the command to use would be -=
<br><br>./bin/kafka-producer-perf-test.sh --brokerinfo broker.list=3D0:loca=
lhost:9092 --messages 100000 --topic test --threads 1 --message-size 200 --=
async --batch-size 200 --delay-btw-batch-ms 6<br>
<br>Also, I do see what you mentioned. The producer throughput numbers,(eve=
n with log.flush.interval=3D1000) is much lower.<br><br>Total Num Messages:=
 100000 bytes: 10000000 in 5.691 secs<br>Messages/sec: 17571.6043<br>MB/sec=
: 1.6758<br>
<br>And, for sending message one by one using this command -<br><br>./bin/k=
afka-producer-perf-test.sh --brokerinfo broker.list=3D0:localhost:9092 --me=
ssages 100000 --topic test --threads 1 --message-size 100<br><br>The number=
s are not much different, just a little lower -<br>
<br>Messages/sec: 15181.4179<br>MB/sec: 1.4478<br><br>There seems to be som=
ething wrong going on inside the Producer, and this will need more investig=
ation.<br><br>Thanks,<br>Neha<br><br><div class=3D"gmail_quote">On Wed, May=
 25, 2011 at 7:09 PM, lurga <span dir=3D"ltr">&lt;<a href=3D"mailto:lurga.l=
e...@gmail.com">lurga....@gmail.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex;">I see, kafka.perf.producer.Producer has a p=
rameter<br>
&quot;batchSize&quot; (default value =3D 200). If I send message one by one=
, I<br>
have to change log.flush.interval.</blockquote></div><br>

--00151747ba967d8b6804a424c3bf--