Load 66MB text file to TStringList

[Vijaya]

I am having trouble loading a big text file into StringList.
I am getting OutofMemory error.

[John Herbster]

"Vijaya" <gudu...@yahoo.com> wrote

> I am having trouble loading a big text file into
> StringList. I am getting OutofMemory error.

I am not surprised. I doubt that it was designed
to handle files that large. What were you going
to going to do with the data once loaded?
Rgds, JohnH

[Mike Williams (TeamB)]

Vijaya wrote:

> I am having trouble loading a big text file into StringList.
> I am getting OutofMemory error.

Which OS? The biggest text file on my system is only 33MB but it fully
loads into a tStringList in less than 2 seconds.

--
-Mike (TeamB)

[Nicholas Sherlock]

Vijaya wrote:
> I am having trouble loading a big text file into StringList.
> I am getting OutofMemory error.

Show your code. 66MB is not an unreasonable memory request (Although a
redesign of your approach is probably in order).

Cheers,
Nicholas Sherlock

--
http://www.sherlocksoftware.org

[Francois Piette [ICS & Midware]]

> I am having trouble loading a big text file into StringList.
> I am getting OutofMemory error.

I don't know what your application is. But probably using a memory mapped
file is a better approach than using a stringlist if you need to handle
large files as a whole. If you want to be able to handle really large files
(gigabytes files), then you should probably even be more clever because you
can't map such large file completely in memory. You'll have to handle it by
chunk of resonable size to fit into memory.

Contribute to the SSL Effort. Visit http://www.overbyte.be/eng/ssl.html
--
francoi...@overbyte.be
Author of ICS (Internet Component Suite, freeware)
Author of MidWare (Multi-tier framework, freeware)
http://www.overbyte.be

[Vijaya]

Actually I am exporting say 400 table data into a Text File, then Zip it. Zip takes about 6MB.

So I can send that Zip file to anyone who want to look at my data in my company.

I have a program which
unzip the file
then load the Text File into a StringList1.
Split the StringList1 into a StringList2(TableNMame, StringListObject3) based on the tableheader in the text file.
Read StringListObject3 and then
build Insert/Update Statments based on whether record exists or not.

[Vijaya]

I had trouble loading TextFile to StringList using TStringList.LoadFile, when Textfile has empty characters in the middle.

So I rewrote my own methods using Borland code like below
------------------
Procedure SetTextStr(const Value: string; plst: TStrings);
var
P, Start: PChar;
S, S2: string;
iLoop, iLen, iPos1, iPos2: integer;
begin
try
pLst.Clear;

P := Addr(Value[1]); // address to the first character in the Value String
iLen := Length(Value);
iLoop := 0;
if P <> nil then
while (iLoop < iLen) do // P^ <> #0 do ;;; this was wht Delphi coded actually in Stringlist.LoadFromFile procedure.
begin
// but it won't work with my backup/restore files which has #0 characters in memo fields.
Start := P; // so VG changed this according to convenience to use in my project.
iPos1 := iLoop;
while not (P^ in [#0, #10, #13]) do
begin
Inc(P);
Inc(iLoop);
end;
iPos2 := iLoop;
iPos2 := iPos2 - iPos1;
try
SetString(S, Start, iPos2);
if (iPos2 <> 0) then // copies characters from START pointer if , number of characters to be copied is > 0
pLst.Add(S)
else
if (S <> #0) then
begin
// VG 04/07/05 to remove the last 2 or 4 junk characters at the end of file.
SetLength(S2, strLen(Start));
try
S2 := String(Start);
if ( Length(Trim(S2)) > 0 ) then
plst.Add(S);
finally
SetLength(S2, 0);
end;
end;
finally
SetLength(S, 0);
end;
if P^ = #13 then
begin
Inc(P);
Inc(iLoop);
end;
if P^ = #10 then
begin
Inc(P);
Inc(iLoop);
end;
While (P^ = #0) and (iLoop < iLen) do // to handle the NULL characters entered after linefeed characters.
begin
Inc(P);
Inc(iLoop);
end;
end;
finally
end;
end;

procedure LoadFromStream(Stream: TStream; pLst: TStrings);
var
Size: Integer;
S: string;
begin
try
Size := Stream.Size - Stream.Position;
SetString(S, nil, Size);
Stream.Read(Pointer(S)^, Size);
SetTextStr(S, pLst);
finally
end;
end;

Procedure LoadFiletoStringLst(strFileName: String; var lstStrings: TStrings);
var
Stream1: TStream;
begin
lstStrings.Clear;
if not FileExists(strFileName) then
exit;
Stream1 := TFileStream.Create(strFileName, fmOpenRead or fmShareDenyWrite);
try
LoadFromStream(Stream1, lstStrings);
finally
Stream1.Free;
end;
end;
---------------------------

[Vijaya]

I am using Windows 2000.
Actually I am copying data from 400 tables into a text file.
Zip it and redistribute it among the collegues who wants to look at my data can restore the file.

Restore utility
Unzips the file.
Reads the text file into a StringList1.
Then splits the StringList1 into StringList2(tablename, Stringlistobject3) based on the TableHeaders in StringList1.
Using StringListobject3 perform either insert/updated.

Below is my code to read a text file into TStringList
-----------------------

[Jens Gruschel]

> I have a program which
> unzip the file
> then load the Text File into a StringList1.
> Split the StringList1 into a StringList2(TableNMame, StringListObject3) based on the tableheader in the text file.
> Read StringListObject3 and then
> build Insert/Update Statments based on whether record exists or not.

Can't you process the file sequentially, building insert/update
statements while reading it? Using a simple TextFile and ReadLn should
require only very few resources.

--
Jens Gruschel
http://www.pegtop.net