gDiagramm
A.D. Fundum
somewhere, but at the moment Hobbes shows no hits. The bug is related
to a graph with unchanged values, e.g. due to a broken device. Values
that probably will fail (all equal):
1.23
1.23
1.23
1.23
While those should work (not all equal):
1.23
1.23
2.34
1.23
---
A.D. Fundum
> 1.23
> 1.23
> 1.23
> 1.23
What's the fastest way to call gDiagramm.EXE hundreths of times in a
row, applying reasonable solutions?
It's fast! But with the current system(cmdline); the number of times
is noticable. I didn't give multiple threads a try yet, just because
there may be other reasonable options available, and I'm not sure how
any caching of OS/2 perhaps already plays some optimizing role here.
I'll include the (translated) section of code below, but that's just
to illustrate about all options are usable. Please keep in mind this
just is about my slowest processing step, so I'm not looking for new
speed records. Are there any reasonable options to make it go faster,
other than giving - let's say - 2 or 3 treads a try (that'll require
a few safety-related adjustments, but that's not the question here)?
/* Today counter=187, so now "hundreths of times" could be max. 374 */
printf("Creating 2x %u graphs...\n",counter);
for (i=1;i<=counter;i++)
{
/* Avoiding gDiagramm's TRAPs here: values not all equal? /*
if (differences[i]>0)
{
/* Basicly Rexx to C, using system(cmdline); twice here */
/* HTML thumbnail, large graph /*
sprintf(buffer,"@gDiagramm %s.TMP %s.FMT > NUL",fund[i].tick,fund[i].tick);
system(buffer);
sprintf(buffer,"@gDiagramm %s.TMP %slarge.FMT > NUL",fund[i].tick,fund[i].tick);
system(buffer);
sprintf(buffer,"%s.FMT",fund[i].tick);
DosForceDelete(buffer);
sprintf(buffer,"%slarge.FMT",fund[i].tick);
DosForceDelete(buffer);
}
sprintf(buffer,"%s.TMP",fund[i].tick);
DosForceDelete(buffer);
}
---
A.D. Fundum
> giving - let's say - 2 or 3 treads a try (that'll require a few
> safety-related adjustments, but that's not the question here)?
I've tried that with 2 threads. It decreased the total execution time
by just 1.5%. Hardly worth mentioning.
Pentium III 500 MHz, 356 files to be created, CPU load about 80-95%,
taking about 4:58 minutes to complete (including about 30 seconds of
overhead in any case). Size of the 356 files: 4.77 MB.
With 2 thread the average CPU load increases by about 5%, but within
a broader range of about 75-99%. As mentoined above just 1.5% faster.
Hence my hope for an OS-related solution w.r.t. calling an external
*.EXE many times. The harddisk is busy all the time, but obviously
(I'll settlle for 2 MB per minute) not extemely busy. I could try a
third thread, but I don't have any reason to expect a sudden gain
in processing speed then. That's still about 4:20 minutes, excluding
the overhead.
---
Peter Flass
Is the external .exe yours? perhaps making all the code a .dll, making
the exe part just a wrapper that calls the dll, and calling the dll from
your code rather than starting the exe would do the trick.
A.D. Fundum
> Pentium III 500 MHz, 356 files to be created, CPU load about 80-95%,
> taking about 4:58 minutes to complete (including about 30 seconds of
> overhead in any case). Size of the 356 files: 4.77 MB.
Fastest one available here: IBM ThinkPad T42p, same data, CPU load
about 97-99%, taking 1:34 minutes. Size of the 356 files: 9.47 MB.
The size of the graphs depends on the size of the desktop work area
and this one has more pixels than the PIII-screen.
Not mentioned earlier: I'm looking for a speed gain controlled by my
app. E.g. "DosStartSession(&cmdline,KeepInCache);". I don't want any
CONFIG.SYS-related tips&trics, use a RAM IFS, embed gDiagramm itself
in my app, and so on.
So far I assume "data transfer rate" is key here, the harddisk isn't
100% busy with processing data (!) and I assume (!) external *.EXE's
are not cached (while unchanged) and are reloaded each time. Right?
Free(ing) memory shouldn't be an issue, having more than enough/2.
Size of the gDiagramm.EXE in use, not counting files it may need:
291328 bytes. Called 356 times, if the above is right. That's worth
98.91 MB (PIII: average of 0.332 MB per second), which is far more
than all of the input-, output- and tempfiles.
---
A.D. Fundum
> Is the external .exe yours?
No, but the source code happens to be available:
http://www.laser.ru/evgen/soft/gDiagramm/index_l.html
I'm still using v0.0.3, but that's probably the same as "v0.03". His
main page is in Russian, so I'm not sure if the original author still
maintains it. Never mind, learning here...
> perhaps making all the code a .dll, making the exe part just a
> wrapper that calls the dll, and calling the dll from your code
> rather than starting the exe would do the trick.
Thanks for the pointer, perhaps the next best (and actually existing!)
way to go...
I'll take a look at this (only having written Rexx DLLs before). I do
consider gDiagramm to be a "known" OS/2 tool, but an additional DLL
doesn't interfere with that.
---
Andreas Buchinger
solution? HPFS/JFS? Or do the .exe load other data from the disk? RAM-IFS would
be my next suggestion and embed the whole source but you do not like that :-)
Regards,
Andi
Ilya Zakharevich
> What's the fastest way to call gDiagramm.EXE hundreths of times in a
> row, applying reasonable solutions?
One needs 3 parts:
a) call using the full path;
b) call using Dos* calls, not POSIX calls;
c) emxload the executable.
With a do-nothing executable, I could make about 100 calls/sec on P5/133MHz.
Hope this helps,
Ilya
P.S. I never profiled ENV vs no-ENV part...
A.D. Fundum
> If the hard disk works the whole time, wouldn't be a reasonable
> disk cache the solution? HPFS/JFS?
All HPFS. I think the main problem is the external *.EXE of 284.5 KB.
I've modified the old version, which no longer removes any temp files
and doesn't generate the final HTML files. It also doesn't include an
improvement like using 2 threads (+1.5%), or Ilya's full path (+2%).
One can store the files in an empty directory and run Grafiek.EXE:
http://home.uni-one.nl/m1/Matrix.ZIP
> Or do the .exe load other data from the disk?
Yes (included in the 1.05 MB example files). But here step 4 is the
problem, and that calls gDiagramm.EXE a lot of times. The gDiagramm
input files are small, and so are the *.PNG output files (YMMVS). In
pseudo CMD.EXE-code, executed max. 187 times:
gDiagramm.EXE %s.TMP %s.FMT > NUL
gDiagramm.EXE %s.TMP %sgroot.FMT > NUL
I think the problem is a "missing feature", (OS-) cache-related. It's
likely that OS/2 removes the gDiagramm.EXE-environment immediately.
Each time.
That's why a *.DLL, or perhap Ilya's third suggestion, may be faster
indeed. No need to reload the external application each time. The
example takes just under 5 minutes to complete here (PIII 500, MHz),
and the size of my data doesn't explain all disk-related traffic.
> RAM-IFS would be my next suggestion and embed the whole source
> but you do not like that :-)
That's right. I'ld like to assume a default setup of the OS, including
generic cache settings. And it's already a difficult part of a complex
works-for-me-construction. Having to have a RAMDISK.SYS doesn't help
then, unless that was available installed by default. I also don't
speak C++, and the problem is survivable without a solution.
Nevertheless Ilya's EMX-solution may be acceptable, as long as that
tool (a) helps significantly, and (b) nowadays comes with the OS.
It's possible to gain a few percentages with other solutions, but more
important is that there's no OS-based solution. I.e. some "application
cache" with DELAY_APP_DROP_FOR_2_SECONDS_IF_<conditions>-flags.
---
A.D. Fundum
> One needs 3 parts:
> a) call using the full path;
Saves about 2% (using 2 thread 1.5%) of the total time.
> b) call using Dos* calls, not POSIX calls;
Tried that. But no result yet, due to an undocumented return code.
No big deal, "HELP SYSx" knew the pointer to the solution.
> c) emxload the executable.
Well, that seems to qualify, because it nowadays ships with the OS.
It looks to be near-perfect (no number of calls) for my demands!
> With a do-nothing executable, I could make about 100 calls/sec on
> P5/133MHz.
Then I expect at least a 50% gain with my do-something one, including
the overhead. All data files are less than 9 MB in total. Loading the
external gDiagramm.EXE 356 times takes about 11*9 MB=99 MB.
I'll still look at the DLL solution, because this type of applications
has some built-in paradox: create a graph quickly, but waist a lot of
time setting up the required files. If you have a lot of files, and a
fast way to create those required files, it no longer is quick using
the *.EXE per file.
I think there's another solution, but I don't prefer that one too:
only create the large graphs, and use yet another tool to create
the matching thumbnails for all large graphs, in one pass.
What I'm not going to do, I don't speak C++, is adjusting gDiagramm
itself. To make that work with wildcards. Trying that showed another
gDiagramm v0.0.3-exception (with "gDiagramm *.TMP *.FMT").
---
A.D. Fundum
>> c) emxload the executable.
> Then I expect at least a 50% gain with my do-something one
Oops... 4:30 minutes... A gain of about 9.4% with this suggestion C,
including the overhead of the other processing steps. But it's the
lowest hanging fruit so far, not requiring any rewrite.
---
Steven Levine
wrote:
Hi,
> Pentium III 500 MHz, 356 files to be created, CPU load about 80-95%,
> taking about 4:58 minutes to complete (including about 30 seconds of
> overhead in any case). Size of the 356 files: 4.77 MB.
As you found, Ilya's suggestions will help a bit with the process load
time.
What will probably almost double the processing rate is to avoid
reading each data file twice.
For those that wish to peek at the sources, the most recent version I
found was at
http://www.laser.ru/evgen/soft/gDiagramm/gDiagramm005.zip
The source is mostly C-ish C++.
Another speed up would be to rewrite main() to support multiple
argument pairs. This will take some tweaks to the C++ code to prevent
leaks, but these are pretty trivial.
One needs to keep in mind that OS/2, unlike some other OSs is not
optimized for fast process creation so it's almost always better to
allow an app to process multiple arguments.
Steven
--
---------------------------------------------------------------------
Steven Levine <ste...@earthlink.bogus.net>
eCS/Warp/DIY etc. www.scoug.com www.ecomstation.com
---------------------------------------------------------------------
Ian Manners
from
http://shatura.laser.ru/evgen/soft/gDiagramm/index.html
or
http://www.os2site.com/sw/graphics/plot/index.html
Cheers
A.D. Fundum
>> perhaps making all the code a .dll
crash gDiagram.DLL. It only works the first time. The file AAA.TMP is
the first processed, unchanged *.TMP file. The second time it fails.
This is with v0.0.3, because a.o. v0.0.5 does produce 0 bytes-output
files. At least with my current input files.
The first graph generated by gDiagram.DLL looks okay. Not speaking
C++, I'll first look for some common initialization bug. I already
saw the second MaxX is 512 instead of 256... :-/
==========
File AAA.TMP, 256 strings
File AAA.TMP, data was read
MinX=0.000000, MaxX=256.000000, MinY=0.590000, MaxY=6.500000
==========
File AAA.TMP, 256 strings
File AAA.TMP, data was read
MinX=0.000000, MaxX=512.000000, MinY=-503247315040006000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000.000000, MaxY=55108865473146400000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000.000000
Exception = c0000005 occurred at EIP = 16478b77.
SYS1808:
The process has stopped. The software diagnostic
code (exception code) is 0005.
==========
10-26-2009 07:55:34 SYS3175 PID 003c TID 0001 Slot 0094
C:\COMM\GD\DLL\GRAFIEK.EXE
c0000005
16478b77
P1=00000001 P2=30303038 P3=XXXXXXXX P4=XXXXXXXX
EAX=30303030 EBX=30302c8d ECX=30302c8d EDX=00000005
ESI=30303030 EDI=00000000
DS=0053 DSACC=f0f3 DSLIM=ffffffff
ES=0053 ESACC=f0f3 ESLIM=ffffffff
FS=150b FSACC=00f3 FSLIM=00000030
GS=0000 GSACC=**** GSLIM=********
CS:EIP=005b:16478b77 CSACC=f0df CSLIM=ffffffff
SS:ESP=0053:00038fd0 SSACC=f0f3 SSLIM=ffffffff
EBP=00038ff0 FLG=00012206
GDIAGRAM.DLL 0001:00038b77
==========
---
A.D. Fundum
Thanks. This may explain a problem with "unclear" version numbers. The
the googled English version of this page stops at v.0.0.5... :-o
I'll give the *.DLL-concept a try with v0.0.3. If the improvement is
significant, I'll try to add several of the mentioned bells&whistles
to v.0.0.7, and perhaps solve a few other SYS3175-problems too:
Crash with unchanged data (horizontal data line)
Crash with "gDiagramm.EXE *.DAT *.FMT"
Crash, gDiagram.DLL-attempt, when called for the second time
---
A.D. Fundum
> What will probably almost double the processing rate is to avoid
> reading each data file twice.
Agreed. I only create it once and re-use it for the second graph. The
data files are small, but gDiagramm.EXE required a call per graph.
> One needs to keep in mind that OS/2, unlike some other OSs is not
> optimized for fast process creation so it's almost always better to
> allow an app to process multiple arguments.
I'm not really familiar with other OSs, but point taken. Wildcards, or
perhaps "gDiagramm.EXE Data.TXT First.FMT Second.FMT"-constructions.
---
A.D. Fundum
All files needed to reproduce the problem with VAC++ 3.08:
http://home.uni-one.nl/m1/Matrix.ZIP
Use gDiagram.CMD instead of the original MAKEFILE. Run Grafiek.EXE to
obtain 64 (!) *.PNG files. Then it crashes, on its way to an estimated
completion time of about 3 minutes (40% faster).
I've already solved another issue. Creating 2 graphs at once (with 2
*.FMT files) failed. Again after 64 *.PNG files, but also each even
one was bogus (with about 4 instead of 256 processed values).
The problem: it crashes in the *.DLL, and I don't know where. I tried
a few other fixes, but then I ran into a C++ related issue. I don't
think it's stack size-related. Increasing that didn't seem to help at
all. I assume it's more likely to be related to initialization/re-use
of a variable.
Debugging a *.DLL, written in C'ish C++, is a little out of my league.
So:
What's causing it to fail now/where (SYS3175, POPUPLOG.OS2 included)
after creating 64 *.PNG-files?
---
Dave Yeo
Are you initializing the DLL? For EMX (and KLIBC) it is calling
_DLL_Initerm() and I think __ctordtorInit (). For Watcom, LibMain(). Not
sure about VAC++ and I quite likely have the above wrong so someone more
knowledgeable should chime in.
I think it also has to uninitialized at the end as well.
Dave
A.D. Fundum
> Are you initializing the DLL?
near line 196 of the *.HPP of v0.0.3. After increasing that to 512,
I'm now getting "regular" error messages w.r.t. a failing *alloc()-
operation after 215 (of the expected 356) *.PNG files. Progress...
FTR: an initilization problem was solved by moving NumRec from some
*.HPP-object to a *.C(PP) global variable, and set it to 0 each time
my *.DLL function is called.
---
A.D. Fundum
> perhaps making all the code a .dll, making the exe part just a
> wrapper that calls the dll, and calling the dll from your code
> rather than starting the exe would do the trick.
Heading for the 1 minute-mark with an IBM ThinkPad T23, using v0.0.7
with some fixes, it's now crashing in libgd's GDDLL.DLL after 259 (of
356) successfully created *.PNG-files. Probably due to memory leaks,
because just adding a few low hanging free()'s helped a lot:
NumRec=0; /* Make it a global variable, (re)set to 0 each call */
...
free(diagramm.databuff); /* free() without a negative side-effect */
free(diagramm.strBuff); /* free() without a negative side-effect */
The result is without any other solution applied. I'll quit here, but
that's because libgd's GDDLL.DLL (and perhaps some use of C++) is too
complicated for me, and I'm not sure its latest v2.0.34 will help me
at all (not to mention other library updates involved). YMWV.
Obviously I'll apply (most of) the other tips&trics. So far 2 graphs
per pass failed, with empty output files or a bogus second graph.
---