Mongodb sharding performance issue

[Benjamin D'Heure]

Hi everyone,

I work with ar sharding replicated structure, and I have a performance issue with the sharding ...

I also posted this issue on the stackoverflow plateform, it's here to see it.

Thanks to anyone can help me to fix it !

[Benjamin D'Heure]

Is there no advice from 10Gen ?

Please answer quickly, my internship ends soon ...

[Asya Kamsky]

There isn't enough information in that post to know why your single shard inserts are that much slower than direct inserts into the replica set, you didn't include any code showing how you are inserting, nor where your mongos process is running.

Your inserts via mongos into a single shard will always be somewhat slower than a single replica set (though it should not be this much slower) since the way to scale via sharding is to have multiple shards.

So please let us know how you are writing (writeConcern, how many threads, etc) and how your mongos processes are set up relative to your applications...

Asya

[Benjamin D'Heure]

I use only 1 thread => with PHP, I parse an array that contains path to the image, and then I store the image in MongoDB using GridFS ...

As writeConcern, I use the majority.

I actived mongoLog::setLevel(mongoLog::all) and mongoLog::setModule(mongoLog::all), but nothing in the log file...

Ressources :

(all servers are VM on a Proxmox plate-form)

3 mongod: 6Go RAM (50% while inserting), 1CPU (99% idle while inserting...)

2 arbiters : 512Mo RAM (100Mo used), 1CPU(100% idle ...)

1 config : 2Go RAM (200Mo used), 1 CPU(100% idle ...)

1 mongos : 512Mo RAM (100Mo used), 1CPU (98%idle while inserting...)

and the code: $mongo->get_GridFS()->storeFile($path, array(), $options)

the simpliest insert it could be ... It's really a key-value model here: MongoID-picture_data

I've no meta, only what MongoDB generates during the storing process.

To connect to the cluster, I use a MySQL table to record configuration like ip, port, priority, RS, ... => if RS only, i build a connection-pool with the name of my alone RS, and the 3 first servers, sorted on priority. 

if sharded => i build the connection-pool with all mongos.

all adresses in connections-pool are formatted like ip:port .

thanks.

[Benjamin D'Heure]

any help ?

[Derick Rethans]

On Mon, 6 May 2013, Benjamin D'Heure wrote:

> I use only 1 thread => with PHP, I parse an array that contains path to the
> image, and then I store the image in MongoDB using GridFS ...
> As writeConcern, I use the majority.
> I actived mongoLog::setLevel(mongoLog::all) and
> mongoLog::setModule(mongoLog::all), but nothing in the log file...

They don't send anything to a log file, but use PHP's error reporting
system. Thwy will throw notices which you will see if you have
display_errors=1 and error_reporting=-1.

> Ressources :
> (all servers are VM on a Proxmox plate-form)
> 3 mongod: 6Go RAM (50% while inserting), 1CPU (99% idle while inserting...)
> 2 arbiters : 512Mo RAM (100Mo used), 1CPU(100% idle ...)
> 1 config : 2Go RAM (200Mo used), 1 CPU(100% idle ...)
> 1 mongos : 512Mo RAM (100Mo used), 1CPU (98%idle while inserting...)
>
> and the code: $mongo->get_GridFS()->storeFile($path, array(), $options)
> the simpliest insert it could be ... It's really a key-value model here:
> MongoID-picture_data
> I've no meta, only what MongoDB generates during the storing process.
>
> To connect to the cluster, I use a MySQL table to record configuration like
> ip, port, priority, RS, ... => if RS only, i build a connection-pool with
> the name of my alone RS, and the 3 first servers, sorted on priority.
> if sharded => i build the connection-pool with all mongos.
> all adresses in connections-pool are formatted like ip:port .

Can you show your connection code please (no abbreviations)?

cheers,
Derick

--
{
website: [ "http://mongodb.org", "http://derickrethans.nl" ],
twitter: [ "@derickr", "@mongodb" ]
}

[Benjamin D'Heure]

here there's the class that contains all the code to interact with MongoDB : pastebin.com_[mongodb-class]

The client directly calls the storefile() method of this class...

after add display_errors=1 and error_reporting=-1 at the top of the class, I have this return : pastebin.com_[returnNotices]

I found anything wrong in these notices ...

[Benjamin D'Heure]

I don't know how to format this reporting for easy reading ...

Le lundi 13 mai 2013 16:56:01 UTC+2, Derick Rethans a écrit :

[Hannes Magnusson]

The 192.168.36.107:20000 server is offline, causing us to constantly
trying to connect to it.

Please try to the 1.4.0 release (released yesterday). It implements a
list of "known unavailable servers" so we don't try to reconnect to
broken servers on every command.

-Hannes

> --
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
>
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-user...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Benjamin D'Heure]

heu ... it's the php driver release you're talking about, isn't it ?

[Hannes Magnusson]

Oh, sorry :)
Yes, I meant the 1.4.0 version of the MongoDB PHP driver :)

-Hannes

[Benjamin D'Heure]

What's the best way to update the driver on a debian ?

re-use: pecl install mongo ?

or another way ?

[Hannes Magnusson]

pecl upgrade mongo

-Hannes

[Benjamin D'Heure]

Thanks.

I'll try it tomorrow at work, and then I'll feed you back...

[Benjamin D'Heure]

Hi, I've issue the following command : pecl upgrade mongo

and now, I've a part of my PHP code that works, but another one that produces a segmentation fault "[notice] child process pid 424 exit signal Segmentation fault (11)"

How could I debug it ?

[Hannes Magnusson]

Could you follow the steps on
https://bugs.php.net/bugs-generating-backtrace.php on how to generate
a backtrace, and submit a bug report in our jira tracker on
https://jira.mongodb.org/browse/PHP please?

If you can include your script that would be great !

-Hannes

On Thu, May 23, 2013 at 12:51 PM, Benjamin D'Heure

[Benjamin D'Heure]

Important!

To get a backtrace with correct information you must have PHP configured with --enable-debug!

>> how can i configure php with that option, I installed php with apache from apt ... so, apt compile it for me ^^

[Hannes Magnusson]

apt-get install php5-dbg, or something similar.

-Hannes

[Benjamin D'Heure]

I see in the doc some references to a 'core' or 'core dump' file, but what is it exactly ?

[Hannes Magnusson]

Its a dump of the what the program is doing at the time it segfaults.
See http://en.wikipedia.org/wiki/Core_dump

When an application segfaults it generally creates a core dump, unless
your ulimit -c settings disallow it.
In any case, running the application under a debugger directly will
achieve the same results (getting the backtrace), so no actual need
for the actual core dump.

A core dump is system specific and therefore useless outside of the
machine it happened on, which is why it doesn't help sending it to
someone else to get the backtrace; the backtrace needs to be generated
on the system the core was dumped.

-Hannes

[Benjamin D'Heure]

ok, thanks.

But there's another issue : I can't start apache2 with the following command : /usr/sbin/apache2 -X

in /var/log/apache2/error.logs ==> (2)No such file or directory: apache2: could not open error log file /etc/apache2/${APACHE_LOG_DIR}/error.log.
Unable to open logs

In the /etc/apache2/apache2.conf, I've done (comment the variable line, and create the second):
#ErrorLog ${APACHE_LOG_DIR}/error.log
ErrorLog /var/log/apache2/error.log

these 2 only lines at each start with the -X option ...

the only way to start the daemon that works, it's through the init.d script...

2013/5/24 Hannes Magnusson <bj...@10gen.com>

You received this message because you are subscribed to a topic in the Google Groups "mongodb-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-user/JsYh6xa0sH8/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to mongodb-user...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--

D'Heure Benjamin

3e Bachelier - Réseaux & Télécoms

 -- Think green, think before printing --

[Hannes Magnusson]

If you try to create a smaller example of your script then you don't
need apache.
Try running the script from the command line (call; php myscript.php).
If that segmentation faults too, you can run it directly from gdb.

-Hannes

[Benjamin D'Heure]

I've something wrong with my code and I add a ticket on Jira : PHP-838

2013/5/24 Hannes Magnusson <bj...@10gen.com>

[Hannes Magnusson]

Thanks!

-Hannes

[Benjamin D'Heure]

How can I get and install your latest patch on the 1.4.0 driver ? ==> latest github commit ...

pecl upgrade mongo ?

Thanks in advance.

2013/5/24 Hannes Magnusson <bj...@10gen.com>

[Benjamin D'Heure]

sorry, I found:

mkdir -p /tmp/gitsrc && cd $_

git clone git://<address on 1.4 branch>

cd mongo-php-driver

phpize

./configure

make && make install

/etc/init.d/apache2 restart

now when I insert 300 photos with gridFS and 2 shards; it takes only 9sec !!

Thank you very much for your help and your rapidity.

good luck ...

2013/5/25 Benjamin D'Heure <sephi...@gmail.com>