M/DB performance under stress.

Matt Gilg

unread,

Apr 8, 2011, 1:28:39 PM4/8/11

to M/DB Community Forum

When I begin to load M/DB with ~100 requests per second from an
external server, my HTTP requests begin to time-out, and M/DB doesn't
return any results. Here are results from the M/DB log, after M/DB
quits working:

-------------------------------------------------------------------------------------------------------------------------------------------------

^%zewdTrace(11504)="MDB request processing ended at 62189,62395"
^%zewdTrace(11505)="62189,62395: string to sign:POST"_
$C(10)_"50.17.191.147"_$C(10)_"/mdb/request.m
gwsi"_
$C(10)_"AWSAccessKeyId=IISOFJIEUSNXMCNAQSSD&Action=Select&SelectExpression=SELECT
%2
0%2A%20from%20Devices%20WHERE%20vtrunk_id%3D
%27sattech2100%27&SignatureMethod=HmacSHA256&

SignatureVersion=2&Timestamp=2011-04-08T17%3A19%3A39.669Z&Version=2009-04-15"
^%zewdTrace(11506)="62189,62395: hash=; signature
rcvd=emNdNx2VwBtj3Pr5GPUAXoQl7P6hg6d3o85khEMED2o=
"
^%zewdTrace(11507)="MDB request processing ended at 62189,62395"
^%zewdTrace(11508)="62189,62395: string to sign:POST"_
$C(10)_"50.17.191.147"_$C(10)_"/mdb/request.m
gwsi"_
$C(10)_"AWSAccessKeyId=IISOFJIEUSNXMCNAQSSD&Action=Select&SelectExpression=SELECT
%2
0%2A%20from%20Devices%20WHERE%20vtrunk_id%3D
%271029346%27&SignatureMethod=HmacSHA256&Sign

atureVersion=2&Timestamp=2011-04-08T17%3A19%3A35.529Z&Version=2009-04-15"
^%zewdTrace(11509)="62189,62395: hash=W0I2w5BhoQWy95/7/SKObd/
ueD9VLqDB7LZgsAjPFG8=; signature rcvd=
W0I2w5BhoQWy95/7/SKObd/ueD9VLqDB7LZgsAjPFG8="
^%zewdTrace(11510)="62189,62395: $$init"
^%zewdTrace(11511)="62189,62395: string to sign:POST"_
$C(10)_"50.17.191.147"_$C(10)_"/mdb/request.m
gwsi"_
$C(10)_"AWSAccessKeyId=IISOFJIEUSNXMCNAQSSD&Action=Select&SelectExpression=SELECT
%2
0%2A%20from%20Devices%20WHERE%20vtrunk_id%3D
%27sattech1400%27&SignatureMethod=HmacSHA256&

SignatureVersion=2&Timestamp=2011-04-08T17%3A19%3A35.768Z&Version=2009-04-15"
^%zewdTrace(11512)="62189,62395: hash=bigNsUGaDK65JrkskG85W9keu/
uCFuXL2Hom4NNRt7A=; signature rcvd=
bigNsUGaDK65JrkskG85W9keu/uCFuXL2Hom4NNRt7A="
^%zewdTrace(11513)="62189,62395: $$init"
^%zewdTrace(11514)="62189,62395: string to sign:POST"_
$C(10)_"50.17.191.147"_$C(10)_"/mdb/request.m
gwsi"_
$C(10)_"AWSAccessKeyId=IISOFJIEUSNXMCNAQSSD&Action=Select&SelectExpression=SELECT
%2
0%2A%20from%20Devices%20WHERE%20vtrunk_id%3D
%271029346%27&SignatureMethod=HmacSHA256&Sign

atureVersion=2&Timestamp=2011-04-08T17%3A19%3A35.791Z&Version=2009-04-15"
^%zewdTrace(11515)="62189,62395:
hash=DIbDIlGqD2D4l2QHqR0cpJZkmkZIM7Iu54MIJ+p6hik=; signature rcvd=
DIbDIlGqD2D4l2QHqR0cpJZkmkZIM7Iu54MIJ+p6hik="
^%zewdTrace(11516)="62189,62395: $$init"
^%zewdTrace(11517)="62189,62395: string to sign:POST"_
$C(10)_"50.17.191.147"_$C(10)_"/mdb/request.m
gwsi"_
$C(10)_"AWSAccessKeyId=IISOFJIEUSNXMCNAQSSD&Action=Select&SelectExpression=SELECT
%2
0%2A%20from%20Devices%20WHERE%20vtrunk_id%3D
%27sattech7777%27&SignatureMethod=HmacSHA256&

SignatureVersion=2&Timestamp=2011-04-08T17%3A19%3A36.391Z&Version=2009-04-15"
^%zewdTrace(11518)="62189,62395:
hash=SUuHtJhWkJ4e2WNy05SIfJl55BG3MycqQ8ffKt1PBew=; signature rcvd=
SUuHtJhWkJ4e2WNy05SIfJl55BG3MycqQ8ffKt1PBew="
^%zewdTrace(11519)="62189,62395: $$init"
^%zewdTrace(11520)="62189,62395: string to sign:POST"_
$C(10)_"50.17.191.147"_$C(10)_"/mdb/request.m
gwsi"_
$C(10)_"AWSAccessKeyId=IISOFJIEUSNXMCNAQSSD&Action=Select&SelectExpression=SELECT
%2
0%2A%20from%20Devices%20WHERE%20vtrunk_id%3D
%27sattech1400%27&SignatureMethod=HmacSHA256&

SignatureVersion=2&Timestamp=2011-04-08T17%3A19%3A35.912Z&Version=2009-04-15"
^%zewdTrace(11521)="62189,62395: hash=T73JC4dlJS2fD/CdyBeinqcWz9eB0VQW/
V7XNRb5DGY=; signature rcvd=
T73JC4dlJS2fD/CdyBeinqcWz9eB0VQW/V7XNRb5DGY="
^%zewdTrace(11522)="62189,62395: $$init"
^%zewdTrace(11523)="62189,62395: string to sign:POST"_
$C(10)_"50.17.191.147"_$C(10)_"/mdb/request.m
gwsi"_
$C(10)_"AWSAccessKeyId=IISOFJIEUSNXMCNAQSSD&Action=Select&SelectExpression=SELECT
%2
0%2A%20from%20Devices%20WHERE%20vtrunk_id%3D
%27sattech1400%27&SignatureMethod=HmacSHA256&

SignatureVersion=2&Timestamp=2011-04-08T17%3A19%3A35.775Z&Version=2009-04-15"
^%zewdTrace(11524)="62189,62395: hash=kP/EYEzl7x8/
VpBf0PDJj5q3BznH77QSnzJFdxFjYvg=; signature rcvd=
kP/EYEzl7x8/VpBf0PDJj5q3BznH77QSnzJFdxFjYvg="
^%zewdTrace(11525)="62189,62395: $$init"
^%zewdTrace(11526)="62189,62395: string to sign:POST"_
$C(10)_"50.17.191.147"_$C(10)_"/mdb/request.m
gwsi"_
$C(10)_"AWSAccessKeyId=IISOFJIEUSNXMCNAQSSD&Action=Select&SelectExpression=SELECT
%2
0%2A%20from%20Devices%20WHERE%20vtrunk_id%3D
%27sattech1500%27&SignatureMethod=HmacSHA256&

SignatureVersion=2&Timestamp=2011-04-08T17%3A19%3A35.504Z&Version=2009-04-15"
^%zewdTrace(11527)="62189,62395: hash=L+u8YXQv1IVo/
p0MvMVtwf16wi0TvAxJB0sDXOEMEY0=; signature rcvd=
L+u8YXQv1IVo/p0MvMVtwf16wi0TvAxJB0sDXOEMEY0="
^%zewdTrace(11528)="62189,62395: $$init"
^%zewdTrace(11529)="62189,62395: string to sign:POST"_
$C(10)_"50.17.191.147"_$C(10)_"/mdb/request.m
gwsi"_
$C(10)_"AWSAccessKeyId=IISOFJIEUSNXMCNAQSSD&Action=Select&SelectExpression=SELECT
%2
0%2A%20from%20Devices%20WHERE%20vtrunk_id%3D
%271029346%27&SignatureMethod=HmacSHA256&Sign

atureVersion=2&Timestamp=2011-04-08T17%3A19%3A35.510Z&Version=2009-04-15"
^%zewdTrace(11530)="62189,62395: hash=WQPsIesobnn0Ub
+GMoJhjSCBEoJxTaEp2YucrS0a1YQ=; signature rcvd=
WQPsIesobnn0Ub+GMoJhjSCBEoJxTaEp2YucrS0a1YQ="

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

The log looks different when M/DB is returning results to my queries.
Also, in some cases, I've noticed the following in TOP:
130 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kjournald
148 root 20 0 4128 1264 1000 S 0.0 0.2 0:00.02 mountall
367 root 20 0 2256 328 196 S 0.0 0.1 0:00.00 dhclient3
427 root 20 0 5572 2156 1744 S 0.0 0.3 0:00.01 sshd
531 root 20 0 2520 904 720 S 0.0 0.1 0:00.03 xinetd
560 root 20 0 5736 2776 1460 S 0.0 0.4 0:00.04 apache2
563 www-data 20 0 0 0 0 Z 0.0 0.0 0:00.01 apache2
<defunct>
564 www-data 20 0 0 0 0 Z 0.0 0.0 0:03.04 apache2
<defunct>
565 www-data 20 0 0 0 0 Z 0.0 0.0 0:02.75 apache2
<defunct>
658 root 20 0 2952 1344 1140 S 0.0 0.2 0:00.01 zmgwsi

I'm not sure what is going on with the defunct apache processes, but
it seems like this could prove to be a critical performance issue; the
same code runs well on SimpleDB.

Has anybody seen anything like this before? Is there a good way to
view which version of MDB I'm running?

-Matt

rtweed

unread,

Apr 22, 2011, 4:22:13 AM4/22/11

to M/DB Community Forum

I'll need to get my colleague Chris to look at this issue.

However it has got me thinking that I should try creating a Node.js
alternative to the Apache + m_apache layer for M/DB. It would be a
cool option to have available and it will be interesting to compare
with respect to performance.

I'll report back on progress in due course.

Rob

> 148 root 20 0 4128 1264 1000S 0.0 0.2 0:00.02 mountall
> 367 root 20 0 2256 328 196S 0.0 0.1 0:00.00 dhclient3
> 427 root 20 0 5572 2156 1744S 0.0 0.3 0:00.01 sshd
> 531 root 20 0 2520 904 720S 0.0 0.1 0:00.03 xinetd
> 560 root 20 0 5736 2776 1460S 0.0 0.4 0:00.04 apache2

> 563 www-data 20 0 0 0 0 Z 0.0 0.0 0:00.01 apache2
> <defunct>
> 564 www-data 20 0 0 0 0 Z 0.0 0.0 0:03.04 apache2
> <defunct>
> 565 www-data 20 0 0 0 0 Z 0.0 0.0 0:02.75 apache2
> <defunct>

> 658 root 20 0 2952 1344 1140S 0.0 0.2 0:00.01 zmgwsi

project2501

unread,

Apr 23, 2011, 9:36:00 AM4/23/11

to M/DB Community Forum

Node.js would definitely be an awesome backend for it.

project2501

unread,

Apr 23, 2011, 9:44:22 AM4/23/11

to M/DB Community Forum

FWIW, it might be interesting to look at MongoDB (which has a Node.js
module)
as a scalable, distributed backend for the query engine.
Its already nosql and bias to JSON.

On Apr 22, 4:22 am, rtweed <rob.tw...@gmail.com> wrote:

rtweed

unread,

Apr 23, 2011, 10:17:25 AM4/23/11

to M/DB Community Forum

Haha you're making that suggestion to the wrong guy! There would
actually be nothing to gain at all by using MongoDB as a substitute
for GT.M for M/DB. GT.M, being based on a global storage engine, is a
universal NoSQL engine (ie much more than just a document database
like Mongo), tried, tested, rock solid. I've already created several
Node.js client interfaces for it (eg node-mwire) and have all the JSON
mappings needed to get data in and out of it in JSON format (again see
node-mwire).

See http://www.mgateway.com/docs/universalNoSQL.pdf
and also https://github.com/robtweed/node-mwire

Ironically enough, George James, the co-author of the universal NoSQL
paper, has created a proof of concept with GT.M emulating MongoDB :-)

Thanks for the suggestion, but no way ;-)

Rob

Matt Gilg

unread,

Apr 27, 2011, 12:54:29 PM4/27/11

to M/DB Community Forum

Hi Rob,

Do you have an ETA on the node.js bindings?

The apache performance is terrible - It is a struggle to perform more
that 20-40 requests per second without apache pegging the CPU and
halting the service of requests - even on a quad-core high-memory
instance in EC2. I think this is likely a bug, and should be fixed
regardless. I have not tried swamping a server with 'unauthorized'
requests to see if an outsider could launch a successful DoS attack.

Node.js seems like a fast, lightweight way to go about doing this, and
I'm excited to give it a shot.

-Matt

Matt Gilg

unread,

Apr 27, 2011, 1:09:27 PM4/27/11

to M/DB Community Forum

I misspoke - rather than 20-40 request per second, I meant between
20-40 "outstanding HTTP requests" at any given time.

-Matt

rtweed

unread,

Apr 27, 2011, 1:25:59 PM4/27/11

to M/DB Community Forum

I don't have an ETA as I'm writing it in my spare time. It's coming
along but the Select, in particular, will be a big conversion job,
particularly having to write it all inside out and upside down for the
async calls to the database.

However, I have a basic ListDomains round-trip working already, albeit
without the signature checking, but it's good enough to fool the Bolso
SDB client. It's VERY fast as you can imagine :-) I don't have any
benchmarks, but 50-100 requests/sec should be a walk in the park for
it. The ListDomains test is completing in the Node.js code in 3-4 ms
right now, though without signature checking. I'll report further
benchmarks as I get further into it.

Unfortunately I just really don't know how long it will take, but I do
want to get it done asap. It's going to be VERY cool.

Cheers

Rob

rtweed

unread,

May 1, 2011, 12:21:11 PM5/1/11

to M/DB Community Forum

Matt

I'm going to be able to put out an early build of node-mdb fairly
soon, but it will be restricted to the following SDB Actions:

CreateDomain
DeleteDomain
ListDomains
DomainMetadata
PutAttributes
GetAttributes
DeleteAttributes

ie it will be missing the Select action (so initially no SQL-like
query capability) and no Batch Actions.

My question for you - would this give you enough to do some useful
testing, benchmarking and stress testing, or will your application
require the missing actions before it's any way useful to you?

I think you'll like this version - it is just amazingly fast and
behaving beautifully so far, so I'd like to let people get an early
look at it to try it out

Cheers

Rob

rtweed

unread,

May 14, 2011, 8:39:30 AM5/14/11

to M/DB Community Forum

Has anyone tried the new Node.js version out yet? Any feedback on
performance/scalability compared with the original Apache/m_apache
based version?

Rob

Reply all

Reply to author

Forward