[U-L] High Volume Batch Processing
Ed Gianni
batch processing, we should use a 3gl (recommended for over 2000 records
and almost mandatory for over 20000 records) . To say the least we were
shocked since we were under the impression we were buying a complete
package. It is especially galling since we have been using a mainframe 4gl
for the last ten years to do virtually all of our online and very high
volume batch processing. Now we are told we must go back to a 3gl. Our
Cobol skills are rusty at best and I can count on one hand those who know C
and/or VB.
Anyway, not that I've vented, I'd like to get any experiences with this
issue as we have a system that will involve significant amounts of high
volume batch processing.
Ed Gianni
National Fuel Gas
Brett Cottrell
It is very hard to say without knowing the exact processing you are trying
to perform and any processing time constraints you have, but to say that if
you need to batch process more than 20,000 records then it has to be done
in 3GL is NOT a correct statement.
On the project I am currently working on we have tables that have 100,000
records and sometimes more, which we batch process with Uniface. I am not
saying that processing 100,000 records is fast, or faster than a 3GL would
be, but processing 100,000 records with any tool is going to take some time.
To optimize the performance of batch processing routines, pay particular
attention to the structure of your Uniface code:
- Process records in batches, clearing out the service in between. I have
found that 100 to 1,000 records seems to work the best, but it is trial and
error.
- If you are processing an entity which is not itself updated, then use
discards to move through the hit-list, use retrieve/x on the target entity
(where possible), process a 'batch' of records, store, commit and clear the
target.
- If you are processing an entity which itself needs to be updated then
process them in PK order, process a 'batch' using setocc to move through
the hit-list, remember the last PK, store, commit, clear and then
re-retrieve where PK >= last PK.
- Watch the transaction size, ie. the number of records you process before
committing. Every record is not efficient, as is every 10,000 records. Try
a commit after processing a batch of records (100-1,000).
- Finally look at the SQL queries that Uniface is generating, and use the
SQL EXPLAIN PLAN to determine where the creation of an index will help.
There are many other minor optimizations than can be used but the above
basic rules give the biggest improvement. Using these 'rules' we have
reduced the run-time of many of our batch processing forms by over half.
Hope this helps,
Brett Cottrell.
KUL...@kellyservices.com
Hi
You dont *have* to have a 3gl program for batch processing. Granted,
something like embedded SQL is probably the fastest option, it is not
necessary. How you design your process has a big effect on how the process
works (sorry for stating the obvious).
It definitely makes sense to do batch processing on the server, not the
client. You will need the Uniface executable for the server for that. If
you are dealing with a RDBMS, you can even use stored procedures to do the
work.
One advantage of using Uniface for batch work is when you are mixing data
sources. We have used Uniface forms for batch processing in the past, and
while the solution may not have been the fastest possble, it had met our
requirements.
If you need more details / references please write to me at
<kul...@kellyservices.com>
Hope this helps
Vivek
Luis M. Martinez
I would add to Brett's tips another one regarding processing an entity
which itself needs to be updated: use paranoid locking (read/lock in READ
trigger) so you save one fetch per record. Using this locking with 100,000
records could make a difference.
Of course if you can't implement this locking because some strange end-user
requirement (normally in batch processing you can), forget me.
Hope this helps,
Luis M. Martinez
IDISA - Mexico
Rob Spahitz
Should you? Probably not!
As you move up through layers (Machine/Assembler/3GL/4GL/-5GL?) things slow
down by design.
As Brett points out, Uniface is slower than 3GL, but can be coded in such a
way that you don't notice or don't care.
Although Uniface has limitations, it can do almost anything that 3GLs can
do. (Funny, I remember the same being said about BASIC vs. 'C' in the old
days. Sometimes BASIC is a better choice, sometimes 'C' is better,
sometimes a mix of both is better. The same goes for Uniface and 3GL.)
--
Rob Spahitz, rspahitz @ maxwell . com
San Diego, CA
Maxwell Technologies/MBS
Home of JAMIS-open, Gov't Accounting Software
Ed Gianni
within Uniface. We'll be sure to pursue them once we get everything
straight with about twenty other issues.
theonees
Your consultants are plain wrong.
Even very heavy batch processing can be done in Uniface.
But it isn't easy. But writing a difficult, heavy batch isn't easy in any
language.
If you don't have experience in this type of Uniface programs:
get somebody who has done this before. It is not important how much that
costs,
you will earn in back.
Some hints are:
- Plan all your IO carefully, and check if it works according to your plan
by looking at explain plans etc. Optimizers can do pretty strange things
every now and then!
- Get your memory management right! This was even more important in Uniface
5.2 but even in Six or Seven you don't want exessive swapping.
- Optimize the size of your transactions. Too many commits slow things
down, but too big transactions need large rollback segements etc wich make
them slower also.
- PAY ATTENTION TO DETAILS. Every tenth of a second wich is repeated 60000
times is 100 minutes extra.
Have a go at it in Uniface first, before you try an other tool,
Good luck,
Theo Neeskens
Ordina Industrie BV
The Netherlands
Ed Gianni <gia...@natfuel.com> wrote in article
<s5a5d3...@natfuel.com>...
> Our consultants have informed us that if we are going to do high volume
> batch processing, we should use a 3gl (recommended for over 2000 records
> and almost mandatory for over 20000 records) . To say the least we were
<snipped>
Voisin
For a big job ( > 1000 or > 2000 records)
We use sql transaction (if it is possible).
The factor is X 100 and we don't have memory problems.
Ed Gianni wrote:
>
> Our consultants have informed us that if we are going to do high volume
> batch processing, we should use a 3gl (recommended for over 2000 records
> and almost mandatory for over 20000 records) . To say the least we were
> shocked since we were under the impression we were buying a complete
> package. It is especially galling since we have been using a mainframe 4gl
> for the last ten years to do virtually all of our online and very high
> volume batch processing. Now we are told we must go back to a 3gl. Our
> Cobol skills are rusty at best and I can count on one hand those who know C
> and/or VB.
>
> Anyway, not that I've vented, I'd like to get any experiences with this
> issue as we have a system that will involve significant amounts of high
> volume batch processing.
>
K. Hairopoulos
What's the presequite software that has to be installed in Win'95
except Solid client and Uniface to connect with Novell?
Has anyone tried this solution? (I suppose that few people are still
running applications in Novell)
Thank you in advance
K. Hairopoulos
ronny
Uniface itself does not support Novell, but if you install Uniface on your
clients and also Solid, then Solid will deal with the communication
between the clients and the Novell server. We tested this configuration
recently, and it works well.
Regards,
Ronny
_________________________
Ronny Horovitz
Formula Technologies
39 Hagalim Boulevard
Herzliya, Israel
Tel: (972) 9 9598749
Fax: (972) 9 9598822
emaIL: ro...@formula.co.il
Pieter Uri
In one way your consultant is right. Some languages are faster for
particular jobs then others. But you must take in account that higher "GL's"
live by the grace of the fast evolution of hardware. If your system is up to
date, you can pull of the job. We've build some processes in batch and it
works fine. Some importent things are:
- Batch-processes must be runned on the server not on a client. Running the
batch at night naturaly gives better performance. Make sure your server
is well configured.
- Investigate your data-model intensifly and make a good search-path
analisys of your retrieves.
Performance tuning on this point is very important. The perfomance of our
datamodel was not sufficient so we've build a second model of our data
that is updated every night. It's very redundant and denormalysed but
perfoms like hell.
- Try to let your database do most of the work by stored procedures.
- Write good proc!!
These are some handvest of making it work. It can be done with good
hardware, good software
and a good datamodel.
good luck.
Pieter Uri
Ed Gianni wrote in message ...
Benoit.L...@acn.etatne.ch
We had many batches to write here, most of them concerning more than 100'000
occ. So maybe I can share a few things.
> - Batch-processes must be runned on the server not on a client. Running
> the
> batch at night naturaly gives better performance. Make sure your server
> is well configured.
>
If a batch is writen in a proper way, you won't save a lot of time : the
bottleneck will then become the database, so there's not much you can do. We
are runing our batches at night on a client. We tried to transfer the batch
on Unix, and it saved less than 20% time.
> - Investigate your data-model intensifly and make a good search-path
> analisys of your retrieves.
> Performance tuning on this point is very important. The perfomance of
> our
> datamodel was not sufficient so we've build a second model of our data
> that is updated every night. It's very redundant and denormalysed but
> perfoms like hell.
>
In the documentation they say that you should draw all your entities
separately and write some more code for the retrieves. It spares time. I
have not tested that point myself.
> - Try to let your database do most of the work by stored procedures.
>
Didn't try.
> - Write good proc!!
>
Yes. You MUST use discards everywhere you can. That way, the batch won't
slow down or overflow your PC's memory (there's a bug with Uniface or
Windows somewhere).
Ex :
retrieve/e "MainEntity"
if ($status < 0) exit
while ($status > 0)
retrieve/e "table"
; use table datas
discard "table", 1, n
; After use, discard all the occurences retrieved,
; even if it's on another form or service, even if this form
; or service is "dropped from memory"
discard "MainEntity", 1 ; not setocc + 1
endwhile
And if we have to update MainEntity, we do it with a subtype. Very
strangely, the first update on the subtype takes a long long time, but it's
okay for the next ones.
We store/commit every 500 occurrences treated.
Also, you can drop indexes on you database if you create occurences.
All this is explained in a Uniface Manual. They also explain how to do a
batch using a report (never tried)
Hope this helps, and I'd like to know if you have more tips.
Benoit.