Memory usage limit in Delphi app
Naoya Suzuki
expanding memory stream".
TotalAddrSpace from GetHeapStatus does not expand from 89128960. I tried to
expand max virtual memory size to 3000 (3G), but it does not effect. Task
Manager is reporting my application uses about
143M bytes of memory, but the Performance Tab said I am using more than 800M
bytes. I think this size need to be close to 2G bytes to complete my app.
Is this a limitation of some kind? Is there any way to increace application
runtime memory size?
any help welcome.
cheers
naoya
Mike Williams (TeamB)
> I have a Delphi app which stops with error message "Out of memory while
> expanding memory stream".
Is your stream growing gradually? There are problems with the memory
manager whenever it has to repeatedly increase the size of a memory
allocation. It does this by allocating new, larger memory, copying the
data, and freeing the original memory. If this happens often enough the
original memory may never get reused. One way around this is to make sure
to allocate a lot of memory all at once in the beginning.
--
-Mike (TeamB)
Naoya Suzuki
However, it is still my question 800Mbytes is a limit for an application
(maybe a Delphi application).
It was just a test in a set of tests (first one) to handle big data. I will
need to handle much bigger data
and up to 10 times. After I changed code acording to your advice, the
application's runtime memory
size became 210Mbytes and Task Manager's Performance Tab showed 430Mbytes in
use. I think
this is not too bad. But I need to test up to 10 times more and I guess
application runtime memory size
would be nearly 2Gbytes. Can I still run this application in this size? If
there is limit, how to change this
limit, so that I can run an application takes more than 800M bytes of
memory.
Regards to everyone in this new group
naoya
"Mike Williams (TeamB)" <mi...@remove.aps-soft.com> wrote in message
news:Xns949C5BA32B...@207.105.83.66...
Mike Williams (TeamB)
> Thank you, Mike. It works better now. However, it is still my
> question 800Mbytes is a limit for an application (maybe a Delphi
> application). It was just a test in a set of tests (first one) to
> handle big data. I will need to handle much bigger data and up to 10
> times. After I changed code acording to your advice, the
> application's runtime memory size became 210Mbytes and Task Manager's
> Performance Tab showed 430Mbytes in use. I think this is not too bad.
> But I need to test up to 10 times more and I guess application
> runtime memory size would be nearly 2Gbytes. Can I still run this
> application in this size? If there is limit, how to change this
> limit, so that I can run an application takes more than 800M bytes of
> memory.
I'm not aware of any such limit that's specific to applications created
with Delphi. I just did a very simple test to allocate almost a GB of
memory. My code just did a SetLength(st, 1000000000) on a string. The
task manager immediately displayed a huge increase in total memory usage
but strangely not a jump in the memory for that specific application.
The statement st := '' immediately made the overall usage go back to
normal. Therefore I'm still not sure that task manager is a good way to
view your program memory usage.
Can you explain a bit of what your program actually does? I'm curious
why it would need that much memory. I sounds like you're working with
very large files.
--
-Mike (TeamB)
Naoya Suzuki
This is a whole sale telco billing software wrtten in Delphi. A supllier
gives us a file contains all customers
usage as well as tariff changed as a flat file format. This program process
this file to validate and sort out
to each of our customers and finally stores to a database. File size is
unknown until we get live data from
supplier, but we can guess few meg bytes to few handred meg bytes. We
recently got new customer
and this is very big one, so that it could be double the size of these files
and I need to test these situation.
Current problem file is 50M bytes and I assume I made 3 times of this size
of data in memory. So that
about 200Mbytes is good size. I need to test up to 500M bytes of file to
process. What I am doing in
program is
1. First, read the file by TMemoryStream to suck up into the memory.
2. Scan through Memory to identify a record and create a class object for
each record.
3. Parse each field in a record to validate and process usage.
4. If all OK, store them to a database. If something wrong some forms can
access all stored object to do something with.
That's all, it is simple, but need to handle huge text file and lots of text
manipulations too. I guess 3 to 4 times of original file size of memory is
used is fair. What do you think?
Regards
naoya
"Mike Williams (TeamB)" <mi...@remove.aps-soft.com> wrote in message
news:Xns949FC68A6C...@207.105.83.66...
Mike Williams (TeamB)
> What I am doing in
> program is
> 1. First, read the file by TMemoryStream to suck up into the memory.
> 2. Scan through Memory to identify a record and create a class object
> for each record.
> 3. Parse each field in a record to validate and process usage.
> 4. If all OK, store them to a database. If something wrong some
> forms can access all stored object to do something with.
> That's all, it is simple, but need to handle huge text file and lots
> of text manipulations too. I guess 3 to 4 times of original file size
> of memory is used is fair. What do you think?
I think there's no need to read the whole file into memory at once.
Since the end goal is to get all of the data into a database I'd simply
open the file as a text file and read one line at a time. For each line
processed parse the data and add a row to the table. You'll still have
to deal with possibly bad data and may even wish to process the file once
just looking for the bad rows. You should be able to handle a file of
any size this way. Trying to allocate a whole lot of memory will simply
cause the OS to start swapping memory to disk. Once it starts thrashing
you'll find that any performance improvement obtained by reading the file
into memory will probably disappear.
--
-Mike (TeamB)
Naoya Suzuki
I understand what you mean. Our existing app is doing that right now and
what I am doing is partially investigate how far we can use memory. Current
one reads a file one line by one line and then goes through 10 passes of
process. Each end of pass, it saves data to a database and read it again
from the database at start of next pass (this means it does that 10 times).
Original app was designed several years ago with Win95 environment and did
not have big memory and files it was processing could be max of a few CDs
size (more than 1G bytes). So they could not design making everything into
memory. Now I was asked how far I can utilirise memory to process whole
things. Current button neck is validation accessing database many time and
I am thinking about some of table could be cached into PC's memory and I
need to determine how much of external data I can store into memory as well.
I started over-night process of 50M bytes file processing, see what happen.
It currently takes 5-6 hours to process without much of optimization in
database. And I am targeting this to come less than 1 hours level. (It
takes less than 1 munits to process 50M bytes file without database access).
naoya
"Mike Williams (TeamB)" <mi...@remove.aps-soft.com> wrote in message
news:Xns949FDBD3AE...@207.105.83.66...
Mark Bracey
allocating 650M because of Out Of Memory errors. In my case, though, it was
the culmination of a lot of little allocations. And this was on a machine
with 2G of main memory.
"Naoya Suzuki" <nsu...@nowhere.com> wrote in message
news:403e...@newsgroups.borland.com...