Large Volume of Data

[Aoli]

I have a large volume of data coming at me and I need to store it as
quickly as possible.

The question will be, how do I tell how much I can take before I get an
out of memory.

I plan to load the incoming into an array until I cannot load more, then
write that set to disk and resume loading into the now erased array.

I want to minimize the disk Put by getting as much into memory as
possible. Sometimes the amount of data will not overflow memory so a
disk Put will not be necessary.

Suggestions on the best way to handle this to get data in and stored
quickly is needed.

I plan to use a data type so I can Put the type in one shot. Is this best ?

Air Code
Private Type tDataType
sData
End type

private atData() as tDataType

atData(0).sData = sName

Put lFNbr, , atData()

[Wendelin Uez]

> I plan to use a data type so I can Put the type in one shot. Is this best
> ?
>
> Air Code
> Private Type tDataType
> sData
> End type
>
> private atData() as tDataType
>
> atData(0).sData = sName
>
> Put lFNbr, , atData()

This is quite good but still has a little bit overhead for file I/O
operations.

In case of speed problems when this overhead becomes important you might
better use Windows' own mechanism of swapping data from RAM to file
employing memory mapped files (MMF) where you can setup a huge disk file and
let Windows transfer data chunks from memory to file in background with high
efficiency, your app only has to fill memory buffer and to tell Windows to
save it.

The only bad thing is that MMFs are of static size, they cannot gow
dynamically, but you can create as many MMFs as your disk can store and
address them as needed.

[ObiWan]

:: On Sun, 8 May 2022 14:57:19 +0200
:: (microsoft.public.vb.general.discussion)
:: <t58fpq$v37$2...@dont-email.me>

:: "Wendelin Uez" <wu...@online.de> wrote:

> In case of speed problems when this overhead becomes important you
> might better use Windows' own mechanism of swapping data from RAM to
> file employing memory mapped files (MMF) where you can setup a huge
> disk file and let Windows transfer data chunks from memory to file in
> background with high efficiency, your app only has to fill memory
> buffer and to tell Windows to save it.
>
> The only bad thing is that MMFs are of static size, they cannot gow
> dynamically, but you can create as many MMFs as your disk can store
> and address them as needed.

Yes, MMF is a way to deal with quickly storing large amounts of data,
in such a case the APIs will deal with memory but the underlying system
mechanism will then take care of asynchronously store the data to the
swapfile (if the MMF is an unnamed one)

Another approach would be using a named pipe; in such a case the
process may create a named pipe and write the received data to it, then
another process or at a later moment, the pipe could be re-read and the
data processed

Just to be clear, here's a possible approach; let's assume the data
comes, over the network, from a listening socket

The main program starts creates a named pipe and opens it in write mode,
next it instances an ActiveX exe which will run as a separate process
passing to it the pipe name, then the main program starts listening on
the socket

The AX exe once started opens the named pipe in read mode and keeps
polling it and waiting for incoming data

The main program receives data on the socket and immediately writes
them to the named pipe

The AX exe sees that there are data in the named pipe, reads them and
processes them; being a separate process and having the named pipe as a
FIFO buffer the AX exe can "spend" the needed time in processing the
data

Notice that the above assumes that the whole thing runs on some
"modern" version of windows (say Windows 7) since in such a case named
pipes can hold up to about 1.5GB of data, that gives plenty of room for
data waiting to be processed, by the way since the AX exe will read
(and remove) data from the pipe, the 1.5GB limit will probably never be
reached

[Wendelin Uez]

> Notice that the above assumes that the whole thing runs on some
> "modern" version of windows (say Windows 7) since in such a case named
> pipes can hold up to about 1.5GB of data, that gives plenty of room for
> data waiting to be processed, by the way since the AX exe will read
> (and remove) data from the pipe, the 1.5GB limit will probably never be
> reached

Oh, didn't know that there is a 1.5GB limit (or a limit at all) for pipes.

But if we talk about 1.5GB or less VB can handle this amount completely in
an own buffer and doesn't need to split data in chunks and manage the
chunks - it can accept and buffer all incoming data in memory without having
it to store into file **before** receiving more data.

[ObiWan]

:: On Mon, 9 May 2022 11:51:26 +0200
:: (microsoft.public.vb.general.discussion)
:: <t5ao85$o1b$3...@dont-email.me>

:: "Wendelin Uez" <wu...@online.de> wrote:

> Oh, didn't know that there is a 1.5GB limit (or a limit at all) for
> pipes.

Maybe I'm wrong, since it comes straight from memory which may be
faulty :) but if I recall it correctly, the limit for the data
"pending" in a pipe is around that value

> But if we talk about 1.5GB or less VB can handle this amount
> completely in an own buffer and doesn't need to split data in chunks
> and manage the chunks - it can accept and buffer all incoming data in
> memory without having it to store into file **before** receiving more
> data.

Sure, but then at some point something should read back that data and
process/store them, using pipes and a separate exe would allow
processing the data flow at max speed w/o having to deal with VB memory
allocation/deallocation and with pretty good efficiency

[Wendelin Uez]

>> Oh, didn't know that there is a 1.5GB limit (or a limit at all) for
>> pipes.
>
> Maybe I'm wrong, since it comes straight from memory which may be
> faulty :) but if I recall it correctly, the limit for the data
> "pending" in a pipe is around that value

Such a 2GB--minus-x - limit might exist if Windows treates pipes simply as
memory buffers what I'm afraid that it does so.

> Sure, but then at some point something should read back that data and
> process/store them, using pipes and a separate exe would allow
> processing the data flow at max speed w/o having to deal with VB memory
> allocation/deallocation and with pretty good efficiency

Yes, I agree. But moving out time expensive jobs in another exe will not
reduce total CPU load but add additional load for management overhead. The
question ist who much, would be an interesting one. If Iwould have to
optimize such huge data pipelining I would run experiments with
multithreading (VB can deal with this) or simply with two different exes.
But I'm retired and out of job :-)

[ObiWan]

:: On Tue, 10 May 2022 12:04:56 +0200
:: (microsoft.public.vb.general.discussion)
:: <t5dir6$8fe$1...@dont-email.me>

:: "Wendelin Uez" <wu...@online.de> wrote:

> Such a 2GB--minus-x - limit might exist if Windows treates pipes
> simply as memory buffers what I'm afraid that it does so.

Again, it's just from the back of my brain, didn't check if the limit
is still in place

> Yes, I agree. But moving out time expensive jobs in another exe will
> not reduce total CPU load but add additional load for management
> overhead. The question ist who much, would be an interesting one. If
> Iwould have to optimize such huge data pipelining I would run
> experiments with multithreading (VB can deal with this) or simply
> with two different exes. But I'm retired and out of job :-)

Well, using a separate exe is exactly what I described, a main exe "A"
which receives the data at full rate from whatever source and feeds
them through a named pipe to a second exe "B" (an ActiveX exe instanced
by "A") which processes it; I know that one may play with
multithreading in VB, but it's somewhat "threading on thin ice" (to
paraphrase an article appeared a lot of time ago on a magazine :D) so I
think that using a separate (activeX) exe may be safer and easier and,
while that will add some overhead, I don't think it will penalize
things so much

[Wendelin Uez]

> so I
> think that using a separate (activeX) exe may be safer and easier and,
> while that will add some overhead, I don't think it will penalize
> things so much

What I never could figure out (because of missing time) is the difference
between an ActiveX exe and a standard exe. I never found satiffying
information about that and therefore always used SendMessage for
communication between two exes in my projects what worked good enough for me
(and perhaps avoided compatibility problems?)

[ObiWan]

:: On Wed, 11 May 2022 12:46:29 +0200
:: (microsoft.public.vb.general.discussion)
:: <t5g97r$aqm$1...@dont-email.me>

:: "Wendelin Uez" <wu...@online.de> wrote:

> What I never could figure out (because of missing time) is the
> difference between an ActiveX exe and a standard exe. I never found
> satiffying information about that and therefore always used
> SendMessage for communication between two exes in my projects what
> worked good enough for me (and perhaps avoided compatibility
> problems?)

An ActiveX exe is an executable which exposes a COM interface,
basically one or more "public class" modules exposing properties and
methods, such an exe can either be run as a standard one (if written to
do so) or only invoked by creating an instance of one of the exposes
classes (interfaces), such an exe will run in its own process (it's an
exe) separated from the process which created the instance, so with
such an approach it's possible to run operations in a separate process
and, using properties and methods, to have a direct interface between
the main process and the AX exe

[Larry Serflaten]

Wendelin Uez wrote:

> What I never could figure out (because of missing time) is the difference

> between an ActiveX exe and a standard exe. I never found satisfying
> information about that ... <snipped for brevity>

As a short primer from years old memory, this may not be technically accurate, but close enough to get the idea across.

In the early years of Microsoft their code base was making use of Dynamically Linked Libraries (DLL) because they afforded the ability to package code in a module that could be shared among different applications. VB could use DLLs by adding references to them in the VB Project. VB was given the ability to create DLLs by adding public code to a BAS module and compiling it up into a DLL. As with all DLLs they were loaded into the program's process memory and ran as part of the single threaded program.

Then they thought; "Well what about shared dialogs, and other user interactions?" Things got a little muddy about how to make that happen, but when the dust settled they came up with executable files that could be referenced like DLLs. They had the ability to show forms and dialogs under the control of a 'host' program, or they could be launched in a stand-alone form if that was appropriate for the task. To accomplish the stand-alone part, the exe had to run in its own process, using its own program memory space. Thus, they were 'executable' files.

At about that time, DLL versioning issues were becoming rampant; some programs ran with this version, some programs ran with that version, leading to the phrase of the day - DLL Hell. To solve the issue Microsoft developed a fix and applied the fix (published Interfaces) across its several products only to market the whole idea as the latest new technology ActiveX.

So, the main difference between an ActiveX and a standard exe is, the ActiveX program will have a public interface you can reference and control from your VB program (DLL style). A standard exe does not (typically) have such an interface.

HTH
LFS

[ObiWan]

:: On Wed, 11 May 2022 15:32:56 +0200
:: (microsoft.public.vb.general.discussion)
:: <20220511153...@mvps.org>

:: ObiWan <obi...@mvps.org> wrote:

> An ActiveX exe is an executable which exposes a COM interface,
> basically one or more "public class" modules exposing properties and
> methods, such an exe can either be run as a standard one (if written
> to do so) or only invoked by creating an instance of one of the
> exposes classes (interfaces), such an exe will run in its own process
> (it's an exe) separated from the process which created the instance,
> so with such an approach it's possible to run operations in a
> separate process and, using properties and methods, to have a direct
> interface between the main process and the AX exe

Forgot, it's also possible to have multiple references to the same AX
exe, all those may point to the same process, but will be separated,
that is, the code will be shared, but the data will be private to each
reference, for example, in VB6 one may issue a "CreateObject" to create
the first instance of an ActiveX exe and then issue whatever number of
"GetObject" to obtain further references to the ActiveX exe, in such a
case the process will be the SAME, but the data belonging to each
reference will be separated; willing to create a data pool to be shared
between different processes it's possible to use a trick, that is an
activeX exe which, upon its very first start, creates an ActiveX DLL
and exposes it through a "get" property, at that point each process
which will get a reference to the ActiveX exe and, through it, to the
DLL will have a shared area represented by the common DLL memory

[ObiWan]

:: On Wed, 11 May 2022 16:23:04 +0200
:: (microsoft.public.vb.general.discussion)
:: <20220511162...@mvps.org>

:: ObiWan <obi...@mvps.org> wrote:

> Forgot, it's also possible to have multiple references to the same AX
> exe, all those may point to the same process, but will be separated,
> that is, the code will be shared, but the data will be private to each

quick example (air code)

Function GetAxExe(sInterface) As Object
Dim oExe As Object

On Local Error Resume Next
Err.Clear
Set oExe = GetObject(,sInterface)
If Err.Number <> 0 Then
Set oExe = CreateObject(sInterface)
End If
Set GetAxExe = oExe
End Function

calling the above a first time with (e.g.)

Set oMyRef = GetAxExe("Excel.Application")

will start Excel (which is an AX exe) and return a reference to its
"Application" interface, if we now call the function again with

Set oMyOtherRef = GetAxExe("Excel.Application")

the returned reference will be obtained from "getobject" and will refer
to the SAME executable process, not to a new one, but the data
belonging to the two references will be separated from each other

[Aoli]

And create an array of AxEXE each in its own memory space operating
simultaneously. I do this for one of my huge apps. that do on screen
processor intensive graphical ops like image fades and other effects
etc. This cannot be done in one main app as there is not enough CPU
allocated to one .EXE.
I pass multiple image filename to each AxEXE and other effects params to
do the image effect and let them rip with ultimate control of the main
app. Works even on my old Win XP Pro laptop.

[ObiWan]

:: On Wed, 11 May 2022 09:34:16 -0700
:: (microsoft.public.vb.general.discussion)
:: <t5goia$7jb$1...@gioia.aioe.org>

:: Aoli <Ao...@Aoli.com> wrote:

> And create an array of AxEXE each in its own memory space operating
> simultaneously. I do this for one of my huge apps. that do on screen
> processor intensive graphical ops like image fades and other effects
> etc. This cannot be done in one main app as there is not enough CPU
> allocated to one .EXE.
> I pass multiple image filename to each AxEXE and other effects params
> to do the image effect and let them rip with ultimate control of the
> main app. Works even on my old Win XP Pro laptop.

Well, create several instances or just get multiple references to a
given one are both allowed, in the first case you'll have multiple
processes so that's a way to run operation "in parallel" using VB6

Then there is a trick which may be used to raise events from an AXexe
or to share data between them and the main process, in both cases the
trick involves using ActiveX DLLs

[Wendelin Uez]

> This cannot be done in one main app as there is not enough CPU
> allocated to one .EXE.

I always thought WIN3.1 had static time slices for each thread, so
that a single thread application can waste less time while waiting
for next time slice if it can control multiple threads simultaniously,
and that later Windows versions (95 or XP?) would have changed
that to more dynamic time slices.

Or in other words, why does a single thread application with heavy
CPU access get less CPU time than the same CPU requests
distributed to multiple threads having additional management
overhead?

Sure, a single thread must be interrupted to allow system processes
and more, but, f.e., why should rendering ten bitmaps in sequence
get less CPU time than ten threads each rendering one bitmap?
The net CPU time of ten threads cannot be less than the same i
nstructions done in one thread. Does the OS really waste so much
time managing the time slices? I know that disk I/O is very slow
and such a thread wastes time while waiting, but is this also true
for waiting for data moving from RAM to CPU and back? Is excuting
a series of CPU commands still much faster than waiting for memory
I/O? If so, I would understand that executing CPU instructions for
memory management is not the limiting factor.