Question about CreateStreamOnHGlobal and GlobalAlloc

[Wenjin Zhang]

According to the documentation, the memory handle must be allocated as
movable and nondiscardable. In our application, we used to follow the
suggestion and allocate memory with GMEM_MOVEABLE. But recently we found our
application doesn't scale well on 8 way machines. CPU usage is always very
low (<=25%). After many hours of debugging, we found the problem is related
to CreateStreamOnHGlobal. My understanding is that there are certainly
limitations for movable global memory. For example, there is not more than
64K items per process, and it seems when a movable memory is lock/unlocked,
Windows needs to lock/unlock the process heap, which creates scalability
problem. We tested changing to allocate memory with GMEM_FIXED instead and
we saw dramatic performance improvement (>%150) on 8 way machines. My
explanation is that after the memory is allocated, using fixed memory
doesn't cause the process heap been lock/unlocked.

To me, the documentation is most likely no longer true, I see no reason why
the memory need to be movable, and nondiscardable/discardable have no
meaning for GlobalAlloc in Win32. We have done quite a lot of tests with
using GMEM_FIXED, everything so far works perfectly well. But still, we are
not follow the document, can someone knows the internals explain why the
handle must be movable, if it's not a documentation issue.

Thanks,
Wenjin

[Ivan Brugiolo [MSFT]]

The Process Heap is locked anyway with and without GMEM_FIXED.
What you are seeing instead is the fact that the heap lock
is used to protect the user-mode handle table as well.

If you are the only user of Memory-Streams and the only caller of
GlobalAlloc in your process,
(BTW, are you building a server with many cross apartment
communication or are you building a UI application wuth a lot of clipboard
operation),
it may work for your application.

I can build a small test case where the lack of GMEM_FIXED
would corrupt the user mode handle-table.

You may considere writing your-own IStream implementation,
if that fits the requirements of your application.

--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

"Wenjin Zhang" <zhan...@hotmail.com> wrote in message
news:e3g6QKBm...@TK2MSFTNGP09.phx.gbl...

[Wenjin Zhang]

Thanks for the reply.

I know the process heap is locked when GlobalAlloc is called, with and
without GMEM_FIXED. Our application doesn't call GlobalAlloc and
CreateStreamOnHGlobal very often, but once we get the IStream pointer, the
application reads the stream (generally there is no write) very frequently,
and often with small sizes. It seems calling IStream->Read causes a critical
section been locked, after the content been copied over, then the critical
section is unlocked. The problem is, it seems that's a global critical
section - one per process. Therefore, even different threads use different
IStream pointers (created on different global memory handles), they are
fighting for the same lock. My guess is that with GMEM_FIXED, IStream->Read
doesn't try to obtain a global lock.

Just roughly looked the API HeapWalk, it seems the GMEM_MOVEABLE is
implemented with the corporation of heap management and module managing the
local handle-table. My guess is that the critical section mentioned above is
the lock of the process heap. When a memory (allocated with GEME_MOVEABLE)
is locked, Windows updates the handle-table, but other than that, it seems
also lock the process heap and update the element flag. Memory allocated
with GMEM_FIXED is allocated from process heap directly, so even the
IStream->Read calls GlobalLock/GlobalUnlock, nothing actually changed, at
least there is no lock/unlock.

Also I don't quite understand why the GMEM_MOVEABLE is still necessary. I
know GlobalRealloc only grow fixed memory in place, but realloc-and-copy
should solve that problem. The description of GetHGlobalFromStream indicates
that the handle could change, so that doesn't break the existing contract.

We could reduce the contention by adding another layer of buffering (for
IStream->Read), but that looks weird because then we buffer memory with
memory.

Wenjin

"Ivan Brugiolo [MSFT]" <ivan...@online.microsoft.com> wrote in message
news:%23B%23AgGCm...@TK2MSFTNGP09.phx.gbl...

[Wenjin Zhang]

Thanks for the reply.

I know the process heap is locked when GlobalAlloc is called, with and
without GMEM_FIXED. Our application doesn't call GlobalAlloc and
CreateStreamOnHGlobal very often, but once we get the IStream pointer, the
application reads the stream (generally there is no write) very frequently,

and often in small sizes. It seems calling IStream->Read causes a critical

section been locked, after the content been copied over, then the critical
section is unlocked. The problem is, it seems that's a global critical
section - one per process. Therefore, even different threads use different
IStream pointers (created on different global memory handles), they are
fighting for the same lock. My guess is that with GMEM_FIXED, IStream->Read
doesn't try to obtain a global lock.

Just roughly looked the API HeapWalk, it seems the GMEM_MOVEABLE is
implemented with the corporation of heap management and module managing the
local handle-table. My guess is that the critical section mentioned above is
the lock of the process heap. When a memory (allocated with GEME_MOVEABLE)
is locked, Windows updates the handle-table, but other than that, it seems
also lock the process heap and update the element flag. Memory allocated
with GMEM_FIXED is allocated from process heap directly, so even the

IStream->Read calls GlobalLock/GlobalUnlock, nothing actually changes, at

least there is no lock/unlock.

Also I don't quite understand why the GMEM_MOVEABLE is still necessary. I
know GlobalRealloc only grow fixed memory in place, but realloc-and-copy
should solve that problem. The description of GetHGlobalFromStream indicates
that the handle could change, so that doesn't break the existing contract.

We could reduce the contention by adding another layer of buffering (for
IStream->Read), but that looks weird because then we buffer memory with
memory.

Wenjin

"Ivan Brugiolo [MSFT]" <ivan...@online.microsoft.com> wrote in message
news:%23B%23AgGCm...@TK2MSFTNGP09.phx.gbl...

[Ivan Brugiolo [MSFT]]

The relevant GlobalAlloc code is something like

LockHeap(hHeap);
hEntry = GetHandle();
p = HeapAlloc(hHeap,HEAP_NO_SERIALIZE,size);
UnlockHeap(hHeap);

while the regular heap code does the locking internalli in ntdll.dll
all the times the HEAP_NO_SERIALIZE flag is NOT specified.

The critical section you are seeing is the lock of the process heap,
(if you look at the address of the critical section and the heap handle,
you should convince yourself of that)
used implicitely or explicitely in many several code-paths.

What does in your applicaiton mandates the use
of the IStream created by the CreateStreamOnHGlobal ?

--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

"Wenjin Zhang" <zhan...@hotmail.com> wrote in message

news:#r64DjCm...@TK2MSFTNGP09.phx.gbl...

[Wenjin Zhang]

The problem we have is not caused by obtaining/releasing lock while calling
GlobalAlloc. We do not call GlobalAlloc very often. Here is our code that
gets the IStream pointer (error checking was removed)

hMem = GlobalAlloc(GMEM_FIXED, ulLength); // we used to use
GEME_MOVEABLE
hr = CreateStreamOnHGlobal(hMem, TRUE, &m_pStream);
hr = m_pStream->Write (bStream, ulLength, &bytesRead);

After initialized the m_pStream, our application access the content very
frequently (mainly read by calling m_pStream->Read). The problem is,

If we allocate memory with GEME_MOVEABLE , it seems m_pStream->Read causes a
critical section (it's a per process lock) been locked and then unlocked,
even different threads are using different m_pStream. We are running tests
on a 8 processor machine, the single critical section basically puts access
to different streams into a queue, so the CPU usage is always very low.

We use IStream created by the CreateStreamOnHGlobal mainly because we want
to have same interface, no matter where the contents come from.

"Ivan Brugiolo [MSFT]" <ivan...@online.microsoft.com> wrote in message

news:ucolnHDm...@TK2MSFTNGP09.phx.gbl...

[Wenjin Zhang]

The problem we have is not caused by obtaining/releasing lock while calling
GlobalAlloc. We do not call GlobalAlloc very often. Here is our code that
gets the IStream pointer (error checking was removed)

hMem = GlobalAlloc(GMEM_FIXED, ulLength); // we used to use
GEME_MOVEABLE
hr = CreateStreamOnHGlobal(hMem, TRUE, &m_pStream);
hr = m_pStream->Write (bStream, ulLength, &bytesRead);

After initialized the m_pStream, our application access the content very
frequently (mainly read by calling m_pStream->Read). The problem is,

If we allocate memory with GEME_MOVEABLE , it seems m_pStream->Read causes a
critical section (it's a per process lock) been locked and then unlocked,
even different threads are using different m_pStream. We are running tests
on a 8 processor machine, the single critical section basically puts access
to different streams into a queue, so the CPU usage is always very low.

We use IStream created by the CreateStreamOnHGlobal mainly because we want
to have same interface, no matter where the contents come from.

"Ivan Brugiolo [MSFT]" <ivan...@online.microsoft.com> wrote in message
news:ucolnHDm...@TK2MSFTNGP09.phx.gbl...

[Carl Daniel [VC++ MVP]]

I can't comment officially (only MSFT can do that), but from what I can
determine, your analysis of the situation is exactly correct. I believe
that the requirement for GMEM_MOVEABLE is a leftover from 16-bit OLE and has
no relevance in a Win32 app.

-cd

[Ivan Brugiolo [MSFT]]

The IStream::Write implementation calls GlobalLock(),
that again takes the Process Heap lock for certain blocks

You should really consider your own implementation of IStream.

--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

"Wenjin Zhang" <zhan...@hotmail.com> wrote in message

news:ulzyXeDm...@TK2MSFTNGP10.phx.gbl...

[Wenjin Zhang]

We may create our own implementation in future releases, I know it's not hard. But....

 

From the description of GlobalAlloc and GlobalLock, it seems GlobalLock will not do anything if the memory is allocated with GMEM_FIXED. Furthermore, it seems IStream::Read also calls GlobalLock(), is that right?

 

Back to my original question, the description of CreateStreamOnHGlobal indicates,

 

hGlobal

[in] Memory handle allocated by the GlobalAlloc function. The handle must be allocated as movable and nondiscardable. If the handle is to be shared between processes, it must also be allocated as shared. New handles should be allocated with a size of zero. If hGlobal is NULL, the CreateStreamOnHGlobal function internally allocates a new shared memory block of size zero.

 

At least part of the description is outdated, my key question is about the second sentence - The handle must be allocated as movable and nondiscardable. nondiscardable is no longer meaningful on Win32, so is the "movable" requirement still valid?

 

Thanks,

Wenjin

 

 

"Ivan Brugiolo [MSFT]" <ivan...@online.microsoft.com> wrote in message news:eIhZ46Dm...@tk2msftngp13.phx.gbl...

[Ivan Brugiolo [MSFT]]

Moveable controls the behavior of the Realloc.
If you mark a global handle as moveable,
the contract is that the underlaying memory block may be moved
(if it's reallocated it can indeed be moved on a different area of the
heap).
If you can guarntee that nobody will use GetHGlobalFromStream,
and nobody will ever reallocate the block, and do some other arbitrary use
of it,
your trick may work. But, it's an explicit violation of the contract of the
API and the IStream, that you will use at your own risk.
So, there are legitimate cases where the moveable flag is expected, needed,
and used.

We would all be happy to deprecate the GlobalXXX and LocalXXX API,
but there's too much existing software that relay on subtle behaviors,
and components in the same process may not interoperate if you change that.

--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

"Wenjin Zhang" <zhan...@hotmail.com> wrote in message

news:#8ZAmXEm...@TK2MSFTNGP10.phx.gbl...

[Wenjin Zhang]

I did some tests and then found that even growing the stream is safe (still
used the GMEM_FIXED). To find out why it works, I set a breakpoint at
GlobalRealloc. When IStream->Write causes the memory to grow, it calls
GlobalRealloc with uFlags set to 0x2002, which is GMEM_DDESHARE |
GMEM_MOVEABLE. Here is the description about the uFlags in MSDN,

GMEM_MOVEABLE
If GMEM_MODIFY is also specified, GMEM_MOVEABLE changes a fixed memory
object to a movable memory object.
If GMEM_MODIFY is not specified, then GMEM_MOVEABLE allows a locked
GMEM_MOVEABLE memory block or a GMEM_FIXED memory block to be moved to a new
fixed location. If neither GMEM_MODIFY nor GMEM_MOVEABLE is set, then fixed
memory blocks and locked movable memory blocks will only be reallocated in
place.

A fix memory is returned because the IStream::Write calls GlobalRealloc with
GMEM_MOVEABLE, which explains why even write is safe.

If my findings are valid on all platforms and are consistent with the source
code, perhaps Microsoft can update the OLE document in the future.

"Ivan Brugiolo [MSFT]" <ivan...@online.microsoft.com> wrote in message

news:ep6QEZGm...@tk2msftngp13.phx.gbl...

[Raymond Chen]

You didn't try testing it with a program that takes the moveable block
handle, reallocs it, and then tries to lock the original handle. A moveable
block is guaranteed never to change its handle even if you realloc it. This
assumption is not valid for fixed blocks. So if you give that program a
fixed block when it expected a moveable block, the program will crash.

"Wenjin Zhang" <zhan...@hotmail.com> wrote in message

news:uXvU3pMm...@tk2msftngp13.phx.gbl...

[Wenjin Zhang]

According to the description of function GetHGlobalFromStream in MSDN,

Remarks
The handle this function returns may be different from the original handle
due to intervening GlobalRealloc calls.

My test app gets back the handle with GetHGlobalFromStream after it grows
the stream, when the memory is initially allocated with GMEM_FIXED,
GetHGlobalFromStream returns a handle different from the one passed to
CreateStreamOnHGlobal, but both are handles of fixed memory.

Wenjin

"Raymond Chen" <http://blogs.gotdotnet.com/raymondc/> wrote in message
news:O6gVnCRm...@TK2MSFTNGP10.phx.gbl...