Speeding up IO

[ruds]

Hi,
I have to work on file which are in GB's. They contain some id's and
their related data. These Id's repeat after few hundred or more lines
but with different data. I want to find these Id's and operate on the
data i.e for each id i want to compare data with similar id down the
file.
For this i'm writting following code...

class Risk
{
static String file;
String str,id,sid="0",eid,s,s1,s2,str1,str2,tok1,tok2;
int
a,i=0,j=0,k,l,sc,line,no,flag=0,flag1=1,flag2=1,start=0,end=0,x,y,cnt;
double
val1=0.0,val2=0.0,val3=0.0,val4=0.0,val5=0.0,val6=0.0,val7=0.0,val8=0.0,val9=0.0,val10=0.0;
FileReader fr,fr1,fr2;
BufferedReader br,br1,br2;
FileWriter fw,fw1,fw2;
StringTokenizer st;
int[] subcase=new int{1,2,3,4};
Vector qid=new Vector();
qid.add(100);
qid.add(101);
qid.add(102);
qid.add(103);
qid.add(104);

double qmax[]=new double[11];
String[][] quad=new String[50][30];

public void compare(String infile) throws IOException
{
cnt=0;
for(i=0;i<qid.size();i++)
{
no=0;
fr=new FileReader(infile);
br=new BufferedReader(fr);
while((str=br.readLine())!=null)
{
no++;
if((str.startsWith("$"))||(str.startsWith("-CONT-")))
continue;
else
{
s2=str.substring(0,10);
if(s2.equals(qid.elementAt(i)))
{
cnt++;
start=no;
end=no+29;
quadarray(infile,start,end);
}
if((cnt==sc) && (i><qid.size()))
{
System.out.println("qid="+qid.elementAt(i));
cnt=0;
writesubcase1();
}
}
}
fr.close();
}

public void quadarray(String ifile,int start,int end) throws
IOException
{
try
{
fr1=new FileReader(ifile);
br1=new BufferedReader(fr1);
line=0;
k=0;
x=0;
while((str1=br1.readLine())!=null)
{
line++;
if((line>=start) && (line<end))
{
if(j==0)
quad[j][k]=str1;
if((k==3) || (k==17) || (k==20))
{
val1=Double.parseDouble(str1.substring(18,36));
if(val1>qmax[x])
{
qmax[x]=val1;
x++;
}
}
if((k==5) || (k==8) || (k==22) || (k==25))
{
val2=Double.parseDouble(str1.substring(54,72));
if(val2>qmax[x])
{
qmax[x]=val2;
x++;
}
}
if((k==11) || (k==14) || (k==28))
{
val3=Double.parseDouble(str1.substring(36,54));
if(val3>qmax[x])
{
qmax[x]=val3;
x++;
}
}
k++;
}
}
}
catch (Exception e)
{ }
}

public void writesubcase1() throws IOException
{
x=0;
try
{
fw=new FileWriter("Result.txt",true);
for(y=0;y<30;y++)
{
if((y==0) || (y==1) || (y==2) || (y==4) || (y==6) || (y==7)
|| (y==9) || (y==10) || (y==12) || (y==13) || (y==15) || (y==16) ||
(y==18) || (y==19) || (y==21) || (y==23) || (y==24) || (y==26) ||
(y==27))
{
fw.write(quad[0][y]+"\n");
continue;
}
else
{
if((y==3) || (y==17) || (y==20))
{
s=quad[0][y];
fw.write(s.substring(0,28)+qmax[x]
+s.substring(37)+"\n");
x++;
continue;
}
if((y==5) || (y==8) || (y==22) || (y==25))
{
s=quad[0][y];
fw.write(s.substring(0,64)+qmax[x]+"\n");
x++;
continue;
}
if((y==11) || (y==14))
{
s=quad[0][y];
fw.write(s.substring(0,46)+qmax[x]
+s.substring(55)+"\n");
x++;
continue;
}
if(y==28)
{
s=quad[0][y];
fw.write(s.substring(0,46)+qmax[x]+"\n");
x++;
break;
}
}
}
fw.close();
}
catch(Exception e)
{}
}

public static void main(String []args)
{
Risk r=new Risk();
file=args[0];
r.compare();
}

Can you tell me how can i speed up IO coz its really taking hell lot
of time to process data.

Thankx in advance.>

[Patricia Shanahan]

ruds wrote:
> Hi,
> I have to work on file which are in GB's. They contain some id's and
> their related data. These Id's repeat after few hundred or more lines
> but with different data. I want to find these Id's and operate on the
> data i.e for each id i want to compare data with similar id down the
> file.
> For this i'm writting following code...

...

>
> Can you tell me how can i speed up IO coz its really taking hell lot
> of time to process data.

...

With no comments, many unexplained constants, and identifiers like
"flag", reading the code is unlikely to be fun.

Can you give an outline of the theory of operation of the program?

Patricia

[a24...@googlemail.com]

On Apr 11, 5:57 am, "ruds" <rudra...@gmail.com> wrote:
> String str,id,sid="0",eid,s,s1,s2,str1,str2,tok1,tok2;
> int
> a,i=0,j=0,k,l,sc,line,no,flag=0,flag1=1,flag2=1,start=0,end=0,x,y,cnt;
> double
> val1=0.0,val2=0.0,val3=0.0,val4=0.0,val5=0.0,val6=0.0,val7=0.0,val8=0.0,val9=0.0,val10=0.0;
> FileReader fr,fr1,fr2;
> BufferedReader br,br1,br2;
> FileWriter fw,fw1,fw2;

For those lines of Java code you would be shot here. I have an idea.
You want to write unmaintainable code? You want to do text processing?
Use Perl.

An additional hint. There is no point in repeating the same question
every third day, ignoring the advice you got. Last time you were
already told to add comments, to explain in some detail what the code
is supposed to do and to clearly structure the code. You couldn't be
bothered. You are clearly a Perl guy.

[Casper]

I can't really read your programs intension or structure but here are a
few pointers:

- Don't use Vector, they have been replaced a long time ago by the
faster ArrayList.
- For gods sake, don't use meaningless variables like that.
- For performance enhancements, try to use the java.nio/memory mapped file.
- Never catch Exception and then do nothing, at the very least throw a
stacktrace.
- Try to use another mechanism than all those if's, a lookup table or
switch would yield better results.
- Use a profiler to find out where the bulk of the time is spent.
- For gods sake, you make it look like Java is procedural and not OO.

/Casper