GridFS memory leak
139 views
Skip to first unread message
Mar65
Nov 1, 2011, 11:54:42 AM11/1/11
to mongodb-user
I've written a program which uses GridFS to store large files. Doing
this the MongoDB server seems to load the files into its memory than
store them but didn't free the used memory. For every new file the
memory used by mongod.exe is growing. This all ends up in the result
that the windows server doesn't respond if there is no more free
memory. Only thing I can do is to switch the machine hard off. I've
tried MongoDB 2.0.1 and 1.8.4 with different parameters (Journal on/
off). No change. I also used MongoVue to store files in the GridFS
which ends in the same result. So for me this looks like a very
serious issue.
mongod.exe is running on Windows Server 2008 R2 as a service.
It has taken some time to figure out what's going on and to find a
work arround, but until now no success. So It would be great if anyone
has an idea how to deal with that issue
Best regards
this the MongoDB server seems to load the files into its memory than
store them but didn't free the used memory. For every new file the
memory used by mongod.exe is growing. This all ends up in the result
that the windows server doesn't respond if there is no more free
memory. Only thing I can do is to switch the machine hard off. I've
tried MongoDB 2.0.1 and 1.8.4 with different parameters (Journal on/
off). No change. I also used MongoVue to store files in the GridFS
which ends in the same result. So for me this looks like a very
serious issue.
mongod.exe is running on Windows Server 2008 R2 as a service.
It has taken some time to figure out what's going on and to find a
work arround, but until now no success. So It would be great if anyone
has an idea how to deal with that issue
Best regards
Eliot Horowitz
Nov 2, 2011, 12:34:59 AM11/2/11
to mongod...@googlegroups.com
How are you measuring?
Remember virtual size will go up with data volume.
Remember virtual size will go up with data volume.
> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>
Mar65
Nov 2, 2011, 4:33:41 AM11/2/11
to mongodb-user
Hi Eliot,
Measuring is easy, Open Windows Task Manager add a file to GridFS.
Example: Add a 100 MB File shows mongod.exe using 100 MB. Add another
100 MB and you see that 200 MB is used by mongod.exe.
This results in the fact that you reach the point where the physical
memoy is completly used which makes the system slow and if all memory
is used the system doesn't respond at all. Which means you aren't able
to simply log in.
I understand that the virtual size grows but nevertheless it coudn't
be that it grows until the system is unsuable. If this is so, there
must be a switch that allows me to limit the used memory and if i
didn't oversee it there is none.
So the simple question is the behaviour as described?
Bye
Volkmar
On 2 Nov., 05:34, Eliot Horowitz <el...@10gen.com> wrote:
> How are you measuring?
> Remember virtual size will go up with data volume.
>
> On Tue, Nov 1, 2011 at 11:54 AM, Mar65
>
>
>
>
>
>
>
> <volkmar.brauckh...@googlemail.com> wrote:
> > I've written a program which usesGridFSto store large files. Doing
> >memoryused by mongod.exe is growing. This all ends up in the result
Measuring is easy, Open Windows Task Manager add a file to GridFS.
Example: Add a 100 MB File shows mongod.exe using 100 MB. Add another
100 MB and you see that 200 MB is used by mongod.exe.
This results in the fact that you reach the point where the physical
memoy is completly used which makes the system slow and if all memory
is used the system doesn't respond at all. Which means you aren't able
to simply log in.
I understand that the virtual size grows but nevertheless it coudn't
be that it grows until the system is unsuable. If this is so, there
must be a switch that allows me to limit the used memory and if i
didn't oversee it there is none.
So the simple question is the behaviour as described?
Bye
Volkmar
On 2 Nov., 05:34, Eliot Horowitz <el...@10gen.com> wrote:
> How are you measuring?
> Remember virtual size will go up with data volume.
>
> On Tue, Nov 1, 2011 at 11:54 AM, Mar65
>
>
>
>
>
>
>
> > I've written a program which usesGridFSto store large files. Doing
> > this the MongoDB server seems to load the files into itsmemorythan
> > store them but didn't free the usedmemory. For every new file the
> >memoryused by mongod.exe is growing. This all ends up in the result
Chris Westin
Nov 2, 2011, 2:25:33 PM11/2/11
to mongodb-user
What you're describing is the correct behavior. MongoDB stores data
in memory mapped files. So, as you store more data, the files grow
larger, and because they are mapped into mongod's address space, the
size of its process grows larger.
However, the operating system should be paging these out. A
process'es virtual address space can be much larger than the amount of
physical ram on the machine, and pages that aren't currently in use
will be written out to disk. If all you're doing is adding files,
you're going to keep requiring more pages to be loaded so they can be
written to. Eventually, you'll use up as much of the physical ram as
the OS will allow, and it will then start writing out older dirty
pages in order to make room for new ones. This could take some time;
if you are monitoring your system, you'll see a lot of writes as
demands for physical ram from other processes cause this process to
evict its working set. If you're starting up new processes, you can
expect that to take some time as the OS writes out the other pages in
order to make them available for new processes.
Chris
in memory mapped files. So, as you store more data, the files grow
larger, and because they are mapped into mongod's address space, the
size of its process grows larger.
However, the operating system should be paging these out. A
process'es virtual address space can be much larger than the amount of
physical ram on the machine, and pages that aren't currently in use
will be written out to disk. If all you're doing is adding files,
you're going to keep requiring more pages to be loaded so they can be
written to. Eventually, you'll use up as much of the physical ram as
the OS will allow, and it will then start writing out older dirty
pages in order to make room for new ones. This could take some time;
if you are monitoring your system, you'll see a lot of writes as
demands for physical ram from other processes cause this process to
evict its working set. If you're starting up new processes, you can
expect that to take some time as the OS writes out the other pages in
order to make them available for new processes.
Chris
Scott Hernandez
Nov 2, 2011, 4:12:52 PM11/2/11
to mongod...@googlegroups.com
While this is generally true of virtual memory managers and processes,
the mongod process flushed dirty data (in the memory mapped files)
periodically (syncDelay option default to 60 seconds) so when the OS
does decide to make more memory available (from mongod) it can do so
from the already clean pages which will not require any writes to
disk.
the mongod process flushed dirty data (in the memory mapped files)
periodically (syncDelay option default to 60 seconds) so when the OS
does decide to make more memory available (from mongod) it can do so
from the already clean pages which will not require any writes to
disk.
Reply all
Reply to author
Forward
0 new messages