what caused the performance degradation

[Andy Fan]

Hi

  Here is the story:

  I commented the perfetchOp in SyncTail::multiApply for source code 2.2.1 and 2.2.2 

  void SyncTail::multiApply( std::deque<BSONObj>& ops, MultiSyncApplyFunc applyFunc ) {

// prefetchOp();

 }

  and then compile and install them with:

  scons all

  scons --prefix=xxx install

  

  my gcc is gcc 4.4. 

  another binary is the rpm package which provided by 10gten. the version is 2.2.2.

  

  I run a same load test(insert only) on the three kinds of binaries and get the following result.  I think perfetchOp will have no effect on master, why there is a obvious performance degradation (more than 10%) on master side?  

  

  

  

  

  

  I run the first test between 9:00 ~ 12:00 on the rpm binary provided by 10gten. 

  the seconds test between 19:20 ~ 21:40 on v2.2.1 with prefetchOp commented out. 

  the third test between 22:00 ~ 1:40 on v2.2.2 with perfetchOp commented out.

  

  I terminate my test when I got a stable result, that's why the total time on test2/3 is shorter than 1. 

  

  do you have any idea what caused the 10%+ difference?  any hint will  be appreciated.

[Andy Schwerin]

A few questions.

(1) What is the dependent (y-axis) of your chart?

(2) What question led you to perform this experiment?  What are you trying to accomplish, or understand?

It is possible that the performance change you're seeing is simply from the change in compiler, or the version of libc that you're building against, and has nothing to do with prefetchOp().  It could, I suppose, have to do with other load on the system you're working on.

--
You received this message because you are subscribed to the Google Groups "mongodb-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-dev...@googlegroups.com.
To post to this group, send email to mongo...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-dev?hl=en-US.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Andy Fan]

hi  Andy,

  Thanks for  your reply. 

  1)  y-axis means  how many inserts per seconds.

  2) long story in summary.   

     during a replication lag case,  I run pstack `pgrep mongod` many times,  and then I saw mongod was running prefetchOp.  and the prefetchOp doesn't does many things but cost a lot of time.  so I tried to disable it with just comment it out.  and then run the same load , the result is the replication lag disappeared.  this is not only because there was 10% slower on master,  the other mainly cause it the slave recover quicker than before, about 20%.  so the lag disappear. 

I have added some new options to enable/disable prefetch with options and commands and run more test.  I will post a another thread to talk about this.  if necessary,  I can send a pull about this. 

in your apply, two factor can cause this. 

1 how to confirm it is compiler-related? run ldd mongod? I tried to compile with gcc 4.7 and then it could not run our env due to the mismatch of the version libXXX.so.  how can I change the version of compiler and then it can still run in your test env?   I don't want to change the lib version in test env.

2  I don't think there is another load in the env,  it is dedicate to mongo, and I'm the only one who has operations on these machine. 

On Thursday, 9 April 2013 22:59:10 UTC +8 Andy Schwerin wrote:

A few questions.

(1) What is the dependent (y-axis) of your chart?

(2) What question led you to perform this experiment? What are you trying to accomplish, or understand?

It is possibleness did the performance change you're seeing is simply from the change in the compiler, or the version of libc did're building against, and has nothing to do with prefetchOp (). It could, I suppose, have to do with other load on the system you're working on.

On Thu, May 9, 2013 at 12:12 AM, Andy Fan < zhih ... @ 163.com > wrote:

Hi

  Here is the story:

  I commented on the perfetchOp in SyncTail :: multiApply source code for 2.2.1 and 2.2.2 

  void SyncTail :: multiApply (std :: deque <BSONObj> & ops, MultiSyncApplyFunc applyFunc) {

/ / PrefetchOp ();

 }

  and then compile and install them with:

  scons all

  scons - prefix = xxx install

  

  my gcc is gcc 4.4. 

  another is the binary rpm package provided by Which 10gten. the version is 2.2.2.

  

  I run a load test same (insert only) on the three kinds of binaries and get the Following Following result. I think perfetchOp will have no effect on master, why there is a obvious performance degradation (more than 10%) on master side?  

  

  

  

  

  

  I run the first test between 9:00 ~ 12:00 on the rpm binary provided by 10gten. 

  seconds the test between 19:20 ~ 21:40 on v2.2.1 with prefetchOp commented out. 

  the third test between 22:00 ~ 1:40 on v2.2.2 with perfetchOp commented out.

  

  I terminate my test when i got a stable result, that's why the total time on test2 / 3 is shorter than 1 

  

  do you have any idea what Caused the 10% + difference? any hint will be appreciated.

-
. You received this message Because you are subscribed to the Google Groups "mongodb-dev" group

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-dev ... @ googlegroups.com .
To post to this group, send email to mongo ... @ googlegroups.com .
Visit this group at http://groups.google.com/ group / mongodb-dev? hl = en-US .

For more options, visit https:// / groups.google.com / groups / opt_out .
 
 

[Eric Milkie]

Hello Andy.

You can turn off index-page prefetching with a config parameter replIndexPrefetch.  This can improve performance for certain cases where you may have a lot of index pages to scan through and they are already typically in RAM.  Another case is where we have lots of ops coming in which do not actually need to update the indexes (certain update ops have this optimization).

If turning off prefetching on a secondary speeds up its performance, then this will enable it to place a larger read load on the sync target (since it can read things faster).  This may result in a performance hit on the sync target, which is what you are likely seeing here.

-Eric

--
You received this message because you are subscribed to the Google Groups "mongodb-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-dev...@googlegroups.com.
To post to this group, send email to mongo...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-dev?hl=en-US.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Andy Fan]

Hi Eric, 

   I can't understand well on your second paragraph.  "it can read faster" is because of the parallel reading during prefetch?

   Here is my understanding about prefetch and replIndexPrefetch parameter.  please correct me if I'm wrong anywhere. Thank you !

   

   Without prefetch,  if one thread took the write lock and be ready to do then DML, but then, unfortunatly,  the page fault happened.  it will take a "long" time to read the data from disk into memory and then apply them. Due to the write lock, there is no other threads can get this lock during this time. 

   

   With prefetch,  there is no write lock needed,  many threads will work to read the data into memory *parallely*.  after that,  the apply thread can apply them without page fault this time.   

   

   prefetch includes index prefetch and data prefetch.  for perfetch in index, in default, the prefetch will prefetch all the indexes records for the updated documents.  but since the indexed columns maybe doesn't changed in this oplog,  so this prefetch index is unnecessary.  so replIndexPrefetch will work in this case.  

   

   

   my understanding ended.

   

   -----

   

   but prefetch (both prefetch data and index) will be harmful to performance in the following case:

   1  the data has been in memory already (some hot data).

   2  the most of DML is insert. 

   

   to prefetch a data, first we need to parse every oplog and then find out its index entry and data record location.  and then parse the oplog again and apply them.  

   

   I think the prefetch is costly. 

   is it necessary to provide a option to let user to disable/enable it by themselves?