Allocating memory / Writing large files
Danny
dynamically allocate memory followed by writing it all out to a file
as fast as possible.
I was thinking about using dynamic arrays, something like:
Dim Buffer() as Byte
and then redimensioning as needed.
This memory will be filled by an external routine to which I will just
pass the address with VarPtr(Buffer(0)).
Writing this out, however, is a different story. I know from past
experience that standard VB file routines (Put) are excruciatingly
slow.
Anyway, I'd appreciate comments on both before I start. APIs based
solutions are not only welcome but probably necessary to get the
efficiency I need. Thanks!
Danny
(You guessed it! ;o) Remove NOSPAMFOR before emailing.)
Tony Proctor
is going to be beforehand, this will automatically make the body of the file
visible in your virtual address space. The fact this avoids using heap
storage (as used by VB objects, arrays, etc) makes it very slick. Also, the
O/S will take care of writing it all out to disk, as efficiently as it can.
You don't need to use VB I/O, or even Win32 I/O.
Tony Proctor
"Danny" <NOSPAMFORd...@yahoo.com> wrote in message
news:41fd6d38...@News.Individual.NET...
Tony Proctor
OpenFile method returns the starting address of the mapped data (which is
what you said you needed). When the CloseFile method is called, or when the
class instance is terminated, the data is automatically flushed back to the
file.
Tony Proctor
'--------- MMapWriteClass.cls -------------------
Option Explicit
Private Declare Function CreateFile Lib "kernel32" Alias "CreateFileA" ( _
ByVal lpFileName As String, ByVal dwDesiredAccess As Long, _
ByVal dwShareMode As Long, ByVal lpSecurityAttributes As Long, _
ByVal dwCreationDisposition As Long, ByVal dwFlagsAndAttributes As Long,
_
ByVal hTemplateFile As Long) As Long
Private Declare Function CreateFileMapping Lib "kernel32" Alias
"CreateFileMappingA" ( _
ByVal hFile As Long, ByVal lpFileMappingAttributes As Long, _
ByVal flProtect As Long, ByVal dwMaximumSizeHigh As Long, _
ByVal dwMaximumSizeLow As Long, ByVal lpName As String) As Long
Private Declare Function MapViewOfFile Lib "kernel32" ( _
ByVal hFileMappingObject As Long, ByVal dwDesiredAccess As Long, _
ByVal dwFileOffsetHigh As Long, ByVal dwFileOffsetLow As Long, _
ByVal dwNumberOfBytesToMap As Long) As Long
Private Declare Function UnmapViewOfFile Lib "kernel32" (ByVal lpBaseAddress
As Long) As Long
Private Declare Function CloseHandle Lib "kernel32" (ByVal hObject As Long)
As Long
Private Declare Function GetLastError Lib "kernel32" () As Long
Const PAGE_READONLY As Long = &H2
Const PAGE_READWRITE As Long = &H4
Const PAGE_WRITECOPY As Long = &H8
Const FILE_MAP_COPY As Long = 1
Const FILE_MAP_WRITE As Long = 2
Const FILE_MAP_READ As Long = 4
Const FILE_MAP_ALL_ACCESS As Long = FILE_MAP_WRITE
Const INVALID_HANDLE_VALUE As Long = -1
Private Const CREATE_ALWAYS = 2
Private Const CREATE_NEW = 1
Private Const OPEN_ALWAYS = 4
Private Const OPEN_EXISTING = 3
Private Const FILE_ATTRIBUTE_NORMAL = &H80
Private Const GENERIC_ALL = &H10000000
Private Const GENERIC_EXECUTE = &H20000000
Private Const GENERIC_READ = &H80000000
Private Const GENERIC_WRITE = &H40000000
Private hFile As Long 'File handle
Private hFileSec As Long 'File Section handle
Private hMap As Long 'Address of mapping
Public Function OpenFile(sFile As String, lMax As Long) As Long
' Creates the specified file (must not exist already), sizes it to the given
nummber of bytes,
' and maps it into the address space. The start address of the mapped area
is returned.
' Make sure we haven't already got a mapping
If hMap <> 0 Then CloseFile
' Open the specified file
hFile = CreateFile(sFile, GENERIC_READ Or GENERIC_WRITE, 0, _
ByVal 0&, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, ByVal 0&)
If hFile = INVALID_HANDLE_VALUE Then
Err.Raise vbObjectError + 1000, , "Failed to open file: " &
Hex(Err.LastDllError)
End If
' Map the body into virtual memory
hFileSec = CreateFileMapping(hFile, 0, PAGE_READWRITE, 0, lMax,
"MappedDataFile")
If hFileSec = 0 Then
Err.Raise vbObjectError + 1001, , "CreateFileMapping failed -
LastError: " & Hex(Err.LastDllError)
End If
hMap = MapViewOfFile(hFileSec, FILE_MAP_WRITE, 0, 0, 0)
If hMap = 0 Then
Err.Raise vbObjectError + 1002, , "MapViewOfFile failed - LastError:
" & Hex(Err.LastDllError)
End If
' Tell caller where the file body was mapped to
OpenFile = hMap
End Function
Public Sub CloseFile()
' Close and unmap the current file (if any)
If hMap <> 0 Then
UnmapViewOfFile hMap
hMap = 0
End If
If hFileSec <> INVALID_HANDLE_VALUE Then
CloseHandle hFileSec
hFileSec = INVALID_HANDLE_VALUE
End If
If hFile <> INVALID_HANDLE_VALUE Then
CloseHandle hFile
hFile = INVALID_HANDLE_VALUE
End If
End Sub
Private Sub Class_Initialize()
hFileSec = INVALID_HANDLE_VALUE
hFile = INVALID_HANDLE_VALUE
End Sub
Private Sub Class_Terminate()
CloseFile
End Sub
'---------------------------------------------------
Tony Proctor
"Tony Proctor" <tony_proctor@aimtechnology_NoMoreSPAM_.com> wrote in message
news:ODhp783B...@TK2MSFTNGP10.phx.gbl...
J French
(Danny) wrote:
<snip>
>Writing this out, however, is a different story. I know from past
>experience that standard VB file routines (Put) are excruciatingly
>slow.
I have not found that problem
- try Dim-ing up a byte array of say 100kb and using Put to write it
to a file opened in Binary mode
Personally I have never seen much speed difference between VB and
Delphi ( which uses the APIs directly )
It is all a matter of the number of disk writes and the size of the
buffer
- huge buffers are stupid as they may cause virtual memory activity
both on filling them and on dumping them to disk
Do a few more tests
- and another tip - if you know the size of the file then pre-extend
it before writing the real data
Danny
I can hardly wait to go off-line and dive in! ;o)
Date: Mon, 31 Jan 2005 11:42:12 -0000
Name: "Tony Proctor" <tony_proctor@aimtechnology_NoMoreSPAM_.com>
>In fact, here's a Class that does pretty much what you want Danny. The
>OpenFile method returns the starting address of the mapped data (which is
>what you said you needed). When the CloseFile method is called, or when the
>class instance is terminated, the data is automatically flushed back to the
>file.
...Code omitted for brevity...
Danny
Name: ere...@nowhere.uk (J French)
>>Writing this out, however, is a different story. I know from past
>>experience that standard VB file routines (Put) are excruciatingly
>>slow.
>
>I have not found that problem
A while back I had to write files of only about a few MB (one byte at
a time) and Put simply took forever. I attributed this to the overhead
being repeated for each byte.
Although, it could have been the VB memory management as Tony
explained.
>Do a few more tests
>- and another tip - if you know the size of the file then pre-extend
>it before writing the real data
Yes, I do know the size beforehand. Anyway, Tony's code looks really
good so I'll give it a go.
J French
(Danny) wrote:
<snip>
>A while back I had to write files of only about a few MB (one byte at
>a time) and Put simply took forever. I attributed this to the overhead
>being repeated for each byte.
Yes well that is because you did not understand what was going on
- you were asking MSDOS to leap around for ONE byte
- MSDOS will take roughly the same time for 1 or 100,000 bytes
>Although, it could have been the VB memory management as Tony
>explained.
It was not
>>Do a few more tests
>>- and another tip - if you know the size of the file then pre-extend
>>it before writing the real data
>Yes, I do know the size beforehand. Anyway, Tony's code looks really
>good so I'll give it a go.
It is good - but it is irresponsible of him to give it to you
Go back and re-write your code so it accesses the disk infrequently
- when you understand how to do that you might want to speed it up
- probably you will not bother
@Tony: Giving Uzis to novices is cruel
Danny
One thing though. I notice that allocated memory is not initialized.
This is not a problem in my case because the external routine will
fill it all, but just out of curiosity and for future reference, what
would be fastest way to clear this memory?
Thanks again!
Date: Mon, 31 Jan 2005 17:26:53 GMT
Name: NOSPAMFORd...@yahoo.com (Danny)
Karl E. Peterson
> That works like magic!!!
>
> One thing though. I notice that allocated memory is not initialized.
I didn't know that. Are you sure?
> This is not a problem in my case because the external routine will
> fill it all, but just out of curiosity and for future reference, what
> would be fastest way to clear this memory?
Probably this:
Private Declare Sub ZeroMemory Lib "kernel32" Alias "RtlZeroMemory" (Destination As
Any, ByVal Length As Long)
--
[Microsoft Basic: 1976-2001, RIP]
Danny
Name: ere...@nowhere.uk (J French)
>>A while back I had to write files of only about a few MB (one byte at
>>a time) and Put simply took forever. I attributed this to the overhead
>>being repeated for each byte.
>
>Yes well that is because you did not understand what was going on
>- you were asking MSDOS to leap around for ONE byte
>- MSDOS will take roughly the same time for 1 or 100,000 bytes
Hello? Anybody home? I just said that!
>>Yes, I do know the size beforehand. Anyway, Tony's code looks really
>>good so I'll give it a go.
>
>It is good - but it is irresponsible of him to give it to you
>
>Go back and re-write your code so it accesses the disk infrequently
>- when you understand how to do that you might want to speed it up
>- probably you will not bother
Why the unprovoked hostility? And totally beside the point too...
> @Tony: Giving Uzis to novices is cruel
I beg your pardon?
Newsflash: Asking exotic questions does not make one a novice. I can
handle the code, thank you very much.
You should take a leaf out of Tony's book on how to be helpful and
civil. However, judging by your above outburst "- probably you will
not bother".
Danny
Name: "Karl E. Peterson" <ka...@mvps.org>
>Danny wrote:
>> That works like magic!!!
>>
>> One thing though. I notice that allocated memory is not initialized.
>
>I didn't know that. Are you sure?
Yes, I just tried it again.
If it makes any difference this is VB6 on a W98 machine.
>> This is not a problem in my case because the external routine will
>> fill it all, but just out of curiosity and for future reference, what
>> would be fastest way to clear this memory?
>
>Probably this:
>
>Private Declare Sub ZeroMemory Lib "kernel32" Alias "RtlZeroMemory" (Destination As
>Any, ByVal Length As Long)
Thank you Karl! Much appreciated!
Karl E. Peterson
> Date: Mon, 31 Jan 2005 18:40:41 +0000 (UTC)
> Name: ere...@nowhere.uk (J French)
>
>>> A while back I had to write files of only about a few MB (one byte
>>> at a time) and Put simply took forever. I attributed this to the
>>> overhead being repeated for each byte.
>>
>> Yes well that is because you did not understand what was going on
>> - you were asking MSDOS to leap around for ONE byte
>> - MSDOS will take roughly the same time for 1 or 100,000 bytes
>
> Hello? Anybody home? I just said that!
>
>>> Yes, I do know the size beforehand. Anyway, Tony's code looks really
>>> good so I'll give it a go.
>>
>> It is good - but it is irresponsible of him to give it to you
>>
>> Go back and re-write your code so it accesses the disk infrequently
>> - when you understand how to do that you might want to speed it up
>> - probably you will not bother
>
> Why the unprovoked hostility? And totally beside the point too...
I'd have to guess its because you blamed the tool, and not its handler. (And, adding
insult to some, you blamed the wrong tool at that.) It's a sensitive area for many
who take pride in using the tool well.
Karl E. Peterson
> Date: Mon, 31 Jan 2005 11:15:58 -0800
> Name: "Karl E. Peterson" <ka...@mvps.org>
>
>> Danny wrote:
>>> That works like magic!!!
>>>
>>> One thing though. I notice that allocated memory is not initialized.
>>
>> I didn't know that. Are you sure?
>
> Yes, I just tried it again.
>
> If it makes any difference this is VB6 on a W98 machine.
Huh. Well, I'll have to try again, I guess. Could be a 9x thing, but likely not.
So, you're creating a brand new file of X length, then mapping it, right? Just want
to be sure to have the same case to try.
>>> This is not a problem in my case because the external routine will
>>> fill it all, but just out of curiosity and for future reference,
>>> what would be fastest way to clear this memory?
>>
>> Probably this:
>>
>> Private Declare Sub ZeroMemory Lib "kernel32" Alias "RtlZeroMemory"
>> (Destination As Any, ByVal Length As Long)
>
> Thank you Karl! Much appreciated!
A pleasure.
Bob O`Bob
> Danny wrote:
>
>>Date: Mon, 31 Jan 2005 11:15:58 -0800
>>Name: "Karl E. Peterson" <ka...@mvps.org>
>>
>>>Danny wrote:
>>>
>>>>That works like magic!!!
>>>>
>>>>One thing though. I notice that allocated memory is not initialized.
>>>
>>>I didn't know that. Are you sure?
>>
>>Yes, I just tried it again.
>>
>>If it makes any difference this is VB6 on a W98 machine.
>
>
> Huh. Well, I'll have to try again, I guess. Could be a 9x thing, but likely not.
I think it may be, actually.
Or the lack of an NT thing, anyway.
Karl E. Peterson
> Karl E. Peterson wrote:
>> Danny wrote:
>>> Karl E. Peterson wrote:
>>>> Danny wrote:
>>>>> One thing though. I notice that allocated memory is not
>>>>> initialized.
>>>>
>>>> I didn't know that. Are you sure?
>>>
>>> Yes, I just tried it again.
>>>
>>> If it makes any difference this is VB6 on a W98 machine.
>>
>> Huh. Well, I'll have to try again, I guess. Could be a 9x thing,
>> but likely not.
>
> I think it may be, actually.
> Or the lack of an NT thing, anyway.
Yeah, right you are. I just tried the following (W2K/SP4):
Private Sub Command1_Click()
Dim mm As New CMapFile
Dim i As Long
Dim sum As Long
mm.MapFileNew "c:\temp.bin", &H10000
For i = 0 To mm.MappedSize - 5 Step 4
sum = sum + mm.GetLng(mm.BaseAddress + i)
Next i
Debug.Print sum
End Sub
Using my CMapFile class (http://vb.mvps.org/samples/MapFile) and it produced a 64K
file full of zeros. So, Danny, if you're still listening, don't bother zero'ing
unless you find yourself on 9x.
Tony Proctor
zero" status. They would then appear to be initialised to 0 as soon as you
touch them.
I'm afraid I don't know much about the W9x systems, but I suspect it's a
pretty big difference in their memory management architecture.
Tony Proctor
"Karl E. Peterson" <ka...@mvps.org> wrote in message
news:eddMth#BFHA...@TK2MSFTNGP09.phx.gbl...
Danny
Name: "Karl E. Peterson" <ka...@mvps.org>
>So, Danny, if you're still listening, don't bother zero'ing
>unless you find yourself on 9x.
Oh, I'm still here... This is all very interesting!
In my particular case it is a W98 machine. However, the external
routine I'm passing the address of this memory to, writes on top of it
all. So whether I zero it or not beforehand is moot.
But this is a type of thing worth keeping in the back of one's mind
for those "spurious" errors when things suddenly brake down for "no
reason". I've been bitten by this too many times which is why I,
first, checked to see if the memory is cleared, and then mentioned W98
on the off chance...
Danny
Name: "Karl E. Peterson" <ka...@mvps.org>
>Danny wrote:
I didn't really blame the tool. I just said I'm aware of its overhead
(it's the top paragraph above) which made it unsuitable for the task
at hand.
There is nothing wrong with using Put for smaller files. On the
contrary, it makes the code more readable than using APIs.
Anyway, storm in a tea pot, I hope...
Karl E. Peterson
> But this is a type of thing worth keeping in the back of one's mind
> for those "spurious" errors when things suddenly brake down for "no
> reason". I've been bitten by this too many times which is why I,
> first, checked to see if the memory is cleared, and then mentioned W98
> on the off chance...
Yeah, I'm thinking of adding it, at least as an option, to my own CMapFile class too.
I wasn't aware that 9x didn't initialize, as I too most often write new data myself.
Karl E. Peterson
>> I'd have to guess its because you blamed the tool, and not its
>> handler. (And, adding insult to some, you blamed the wrong tool at
>> that.) It's a sensitive area for many who take pride in using the
>> tool well.
>
> I didn't really blame the tool. I just said I'm aware of its overhead
> (it's the top paragraph above) which made it unsuitable for the task
> at hand.
Well, folks just bristle when they *perceieve* that's the case.
> There is nothing wrong with using Put for smaller files. On the
> contrary, it makes the code more readable than using APIs.
I agree. I'll use Put in most circumstances. It's rare that using the API is
justified, but when it is, it most certainly is!
> Anyway, storm in a tea pot, I hope...
Yep.
J French
(Danny) wrote:
<snip>
>>I'd have to guess its because you blamed the tool, and not its handler. (And, adding
>>insult to some, you blamed the wrong tool at that.) It's a sensitive area for many
>>who take pride in using the tool well.
>
>I didn't really blame the tool. I just said I'm aware of its overhead
>(it's the top paragraph above) which made it unsuitable for the task
>at hand.
>
>There is nothing wrong with using Put for smaller files. On the
>contrary, it makes the code more readable than using APIs.
Put is fine for large files
The trick is to cut down on the number of Disk Accesses
Even the APIs are not much faster than the native VB Put
Even though Windows and modern hardware has a heck of a lot of
caching, /any/ disk activity is inherently slow because it is asking
the OS to jump through a load of hoops.
Here is a Class for writing to a buffered file :-
VERSION 1.0 CLASS
BEGIN
MultiUse = -1 'True
END
Attribute VB_Name = "cWriteFileStream"
Attribute VB_GlobalNameSpace = False
Attribute VB_Creatable = True
Attribute VB_PredeclaredId = False
Attribute VB_Exposed = False
Option Explicit
Private Type TCMN
BufferLen As Long
FileName As String
Buffer As String
BufferPos As Long
WritePos As Long
Channel As Integer
End Type
Private cmn As TCMN
' ---
Private Sub Class_Initialize()
cmn.BufferLen = 100000
End Sub
' ---
Public Sub Create(FileName$)
cmn.FileName = FileName
cmn.Buffer = Space$(cmn.BufferLen)
cmn.BufferPos = 0
cmn.WritePos = 1
cmn.Channel = 0
' ---
cmn.Channel = FreeFile
' --- Zap the file
Open FileName For Output As #cmn.Channel
Close #cmn.Channel
' --- Open in Binary Mode
Open FileName For Binary Access Write As #cmn.Channel
End Sub
' ###########################################################
'
' Write a CrLf terminated line
'
Public Sub WriteLine(L$)
If cmn.Channel = 0 Then
MsgBox "cWriteFileStream - WriteLine - but file not Open"
Exit Sub
End If
Me.WriteBlock L$
Me.WriteBlock vbCrLf
End Sub
' ###########################################################
'
' Write a block of data
'
Public Sub WriteBlock(L$)
Dim Bytes&
If cmn.Channel = 0 Then
MsgBox "cWriteFileStream - WriteBlock - but file not Open"
Exit Sub
End If
Bytes& = Len(L$)
' ---
If cmn.BufferPos + Bytes > Len(cmn.Buffer$) Then
Call LS_FlushBuffer
End If
' --- Catch a long string - the buffer will have been flushed
If Bytes > Len(cmn.Buffer) Then
cmn.Buffer = Space$(Bytes)
End If
Mid$(cmn.Buffer, cmn.BufferPos + 1, Bytes) = L$
cmn.BufferPos = cmn.BufferPos + Bytes
End Sub
' ###########################################################
'
'
Private Sub LS_FlushBuffer()
If cmn.BufferPos > 0 Then
Put #cmn.Channel, cmn.WritePos, Left$(cmn.Buffer,
cmn.BufferPos)
cmn.WritePos = cmn.WritePos + cmn.BufferPos
cmn.BufferPos = 0
End If
End Sub
' ###########################################################
'
'
Public Sub Flush()
If cmn.Channel <> 0 Then
Call LS_FlushBuffer
End If
End Sub
' ###########################################################
'
'
Public Function Free()
If cmn.Channel <> 0 Then
Call LS_FlushBuffer
Close #cmn.Channel
cmn.Channel = 0
End If
End Function
Private Sub Class_Terminate()
Me.Free
End Sub
Danny
Name: ere...@nowhere.uk (J French)
>>There is nothing wrong with using Put for smaller files. On the
>>contrary, it makes the code more readable than using APIs.
>
>Put is fine for large files
>
>The trick is to cut down on the number of Disk Accesses
... code omitted for brevity ...
Indeed! But in order to do that you have to jump through a lot of
hoops (as your code demonstrated). I could've also defined a large UDT
to serve as a buffer, (mem)copy a chunk of bytes and write that out in
one big gulp. There are many possible workarounds.
However, by comparison, using Tony's API solution is not only shorter
but very elegant and more readable too.
Which approach (out of many) is chosen in the end depends on the
circumstances. "Horses for courses", as they say.
BTW, you may be interested to hear that using Tony's "Uzi" this,
erm... "novice" was able to mow down the problem in no time! ;o)
Danny
(You guessed it! Remove NOSPAMFOR before emailing.)
J French
(Danny) wrote:
>Date: Wed, 2 Feb 2005 08:50:13 +0000 (UTC)
>Name: ere...@nowhere.uk (J French)
>
>>>There is nothing wrong with using Put for smaller files. On the
>>>contrary, it makes the code more readable than using APIs.
>>
>>Put is fine for large files
>>
>>The trick is to cut down on the number of Disk Accesses
>... code omitted for brevity ...
>Indeed! But in order to do that you have to jump through a lot of
>hoops (as your code demonstrated).
I must confess I found that code rather simple
>I could've also defined a large UDT
>to serve as a buffer, (mem)copy a chunk of bytes and write that out in
>one big gulp. There are many possible workarounds.
>However, by comparison, using Tony's API solution is not only shorter
>but very elegant and more readable too.
>
>Which approach (out of many) is chosen in the end depends on the
>circumstances. "Horses for courses", as they say.
Yes I totally agree
>BTW, you may be interested to hear that using Tony's "Uzi" this,
>erm... "novice" was able to mow down the problem in no time! ;o)
Now you need to 'package' the code so that it is re-useable
- always handy having 'black boxed' library routines in ones toolkit
Danny
Name: ere...@nowhere.uk (J French)
>I must confess I found that code rather simple
Simple - elegant, let's not split hairs... ;o)
But, I must admit, I also often take the "why simple when it can be
complicated approach" ;o) when I have the time, that is.
It's not only fun, but it has a very serious side as well because it
pushes against the edges of a tool/language and, like fractals, that's
where the most interesting stuff happens.
Doing that facilitates *thorough* learning and comprehensive
understanding, as opposed to just getting by. Because of that, I not
only read your code but I also filed it for future reference.
J French
(Danny) wrote:
>Date: Thu, 3 Feb 2005 08:57:45 +0000 (UTC)
>Name: ere...@nowhere.uk (J French)
>
>>I must confess I found that code rather simple
>
>Simple - elegant, let's not split hairs... ;o)
>
>But, I must admit, I also often take the "why simple when it can be
>complicated approach" ;o) when I have the time, that is.
Phew - I would never do that
Most things that are complicated are ill designed - IME
>It's not only fun, but it has a very serious side as well because it
>pushes against the edges of a tool/language and, like fractals, that's
>where the most interesting stuff happens.
>Doing that facilitates *thorough* learning and comprehensive
>understanding, as opposed to just getting by. Because of that, I not
>only read your code but I also filed it for future reference.
Thoroughly understanding what one is doing is essential, otherwise one
spends more time chasing down problems than coding.
Danny
Name: ere...@nowhere.uk (J French)
>On Thu, 03 Feb 2005 17:12:52 GMT, NOSPAMFORd...@yahoo.com
>(Danny) wrote:
>
>>>I must confess I found that code rather simple
>>
>>Simple - elegant, let's not split hairs... ;o)
>>
>>But, I must admit, I also often take the "why simple when it can be
>>complicated approach" ;o) when I have the time, that is.
>
>Phew - I would never do that
>Most things that are complicated are ill designed - IME
That's a different category. That's what I would call "messy".
What I meant above was empirically probing the limits of the paradigm
in order to comprehend it fully and facilitate learning.
>Thoroughly understanding what one is doing is essential, otherwise one
>spends more time chasing down problems than coding.
Exactly!
Tony Proctor
thread Sam. In there, the initial file is created by the memory mapping,
rather than the norm of mapping an existing file of non-zero length. The
issue is whether the new virgin contents of the virtual memory (and hence
the file when the data is paged back) are random or 0
Tony Proctor
"Sam Hobbs" <sam...@social.rr.com_change_social_to_socal> wrote in message
news:e#iDLCqEF...@tk2msftngp13.phx.gbl...
> "Danny" <NOSPAMFORd...@yahoo.com> wrote in message
> news:41fe82e6...@News.Individual.NET...
> >
> > One thing though. I notice that allocated memory is not initialized.
> > This is not a problem in my case because the external routine will
> > fill it all, but just out of curiosity and for future reference, what
> > would be fastest way to clear this memory?
>
> I am not sure I understand, but if I am the only one that does not
> understand, then I don't need to understand.
>
> Is this allocated memory that is part of a memory-mapped file? Or is it
> memory allocated separately from a memory-mapped file? If it is part of a
> memory-mapped file, then I thought that memory is pre-allocated and
depends
> on what is in the file.
>
>
Sam Hobbs
It is another possible solution. I assume that the corresponding ReadFile
would be easy enough to write.
I am not suggesting this would work in your situation. I don't know your
requirements well enough to know what is the best solution. For the benefit
ot others and for your future reference, that article is worth knowing
about.
"Danny" <NOSPAMFORd...@yahoo.com> wrote in message
news:41fd6d38...@News.Individual.NET...
> I'm working with ~100 MB files and need the most efficient way to
> dynamically allocate memory followed by writing it all out to a file
> as fast as possible.
>
> I was thinking about using dynamic arrays, something like:
> Dim Buffer() as Byte
> and then redimensioning as needed.
>
> This memory will be filled by an external routine to which I will just
> pass the address with VarPtr(Buffer(0)).
>
> Writing this out, however, is a different story. I know from past
> experience that standard VB file routines (Put) are excruciatingly
> slow.
>
> Anyway, I'd appreciate comments on both before I start. APIs based
> solutions are not only welcome but probably necessary to get the
> efficiency I need. Thanks!
>
Sam Hobbs
news:eb6DnNEC...@tk2msftngp13.phx.gbl...
> On systems with an NT kernel, the new pages should be created with a
> "demand
> zero" status. They would then appear to be initialised to 0 as soon as you
> touch them.
What is the difference between "appear to be initialised" and actually
initialised?
Sam Hobbs
> Date: Mon, 31 Jan 2005 18:40:41 +0000 (UTC)
> Name: ere...@nowhere.uk (J French)
>
>>>A while back I had to write files of only about a few MB (one byte at
>>>a time) and Put simply took forever. I attributed this to the overhead
>>>being repeated for each byte.
>>
>>Yes well that is because you did not understand what was going on
>>- you were asking MSDOS to leap around for ONE byte
>>- MSDOS will take roughly the same time for 1 or 100,000 bytes
>
> Hello? Anybody home? I just said that!
Except what you said is quite confusing. First you seem to have indicated
that Put is the problem, then you say that it might be "the overhead being
repeated for each byte". Then you say (in a portion of the message not shown
here) that the problem might be "VB memory management". So perhaps in this
situation a better response would have resulted from a review of the prior
correspondences.
I say this because of the misunderstandings we had in a different thread.
> Why the unprovoked hostility? And totally beside the point too...
Yes the other comments were stronger than necessary, but as I indicated in
another reply, I understand the frustration. Perhaps it would help to
interpret the remarks as being an expression of frustration, not hostility.
In particular, it is probably best to avoid saying things such as "totally
beside the point" to try to dismiss the conversation; that is the type of
thing that just makes things worse.
You can simply say that you have a solution and your are going to use it.
Others can say they disagree that it is the best solution but they should
not continue to criticize it. I have definitely encountered people that have
criticized my chosen solution and those conversations can persist forever.
Sam Hobbs
news:41fe7556...@news.btclick.com...
>
> Go back and re-write your code so it accesses the disk infrequently
> - when you understand how to do that you might want to speed it up
> - probably you will not bother
J French, I think I have been in a similar situation more than a few times
in which someone uses the first solution provided to them and then stick
with it. In this situation, I don't know if Tony's code is a good solution
for Danny's requirements and I won't try to determine if it is.
The above comments could have been stated in many other ways and perhaps it
was not necessary to be as direct as they were, but I suspect I might have
said something similar enough that I hope you understand that I understand
your point of view also.
I think one problem here is that the requirements are not clear enough. This
might be an example of misunderstandings resulting from misunderstood
requirements.
Tony Proctor
that, they are merely a special setting in a page-table entry. When touched,
the O/S virtual memory management allocates a physical memory page and sets
it, and only it, to 0 then.
Tony Proctor
"Sam Hobbs" <sam...@social.rr.com_change_social_to_socal> wrote in message
news:usMg7#pEFHA...@TK2MSFTNGP09.phx.gbl...
Sam Hobbs
solution, but if I were to have suggested a solution, I would have first
asked for more information.
"Danny" <NOSPAMFORd...@yahoo.com> wrote in message
Sam Hobbs
Danny
Name: "Sam Hobbs" <sam...@social.rr.com_change_social_to_socal>
>> One thing though. I notice that allocated memory is not initialized.
>> This is not a problem in my case because the external routine will
>> fill it all, but just out of curiosity and for future reference, what
>> would be fastest way to clear this memory?
>
>I am not sure I understand, but if I am the only one that does not
>understand, then I don't need to understand.
>
>Is this allocated memory that is part of a memory-mapped file? Or is it
>memory allocated separately from a memory-mapped file? If it is part of a
>memory-mapped file, then I thought that memory is pre-allocated and depends
>on what is in the file.
It's part of a memory-mapped file. Since this is a new file there was
nothing in the file before. Therefore, I expected the OS would clear
this memory before making it available, but it doesn't.
As I suspected this is because of W98. Subsequent OSes clear this
memory but W98 doesn't.
Later in the thread others have confirmed this (it's an old thread).
Danny
(memory mapping was the preferred choice).
I know how to write a file, the question was to find the most
efficient method in order to cope with ~100MB files.
Date: Mon, 14 Feb 2005 06:42:08 -0800
Name: "Sam Hobbs" <sam...@social.rr.com_change_social_to_socal>
>See KB article 165942 ("HOWTO: Write Data to a File Using WriteFile API").
Sam Hobbs
Hopefully you understood what he meant, but I wanted to be sure. The
technical details are not important. What is important is whether the
allocated memory is part of a memory-mapped file or memory allocated
separately from a memory-mapped file.
"Tony Proctor" <tony_proctor@aimtechnology_NoMoreSPAM_.com> wrote in message
news:uZW8MvqE...@TK2MSFTNGP10.phx.gbl...
Sam Hobbs
"Tony Proctor" <tony_proctor@aimtechnology_NoMoreSPAM_.com> wrote in message
news:uDKHCtqE...@tk2msftngp13.phx.gbl...
Sam Hobbs
> Date: Mon, 14 Feb 2005 06:22:00 -0800
>
> nothing in the file before. Therefore, I expected the OS would clear
> this memory before making it available, but it doesn't.
>
> As I suspected this is because of W98. Subsequent OSes clear this
> memory but W98 doesn't.
I am not familiar with the details of memory-mapped files. My guess would
have been that the memory would not be initialized, just as memory in a disk
file is not initialized. So I suppose that is why I was confused.
I truly would not have expected the OS to clear the memory automatically,
but it is good that it does.
Sam Hobbs
such. However thre is a reason why that KB article was written and I wanted
to let you and others know about it.
Yes, I knew you had a soution, but as best as I remember, you still had some
lingering questions for the future.
I am sorry if I implied you were unable to figure it out yourself; I did not
intend to.
"Danny" <NOSPAMFORd...@yahoo.com> wrote in message
news:4210d64...@News.Individual.NET...