Using more than 2gb RAM?
Sam Marrocco
of ram in a single process/task. Can anyone suggest any windows-based
alternatives that would work?
I've heard that Windows Server DataCenter can address more than 2gb per
process, but can't--for the life of me--seem to locate anyone who knows
anything in detail about it, or can sell it to me! MS's web site is all
marketing, so it doesn't help me much.
Any other 32-bit alternatives in windows I should be looking into?
--
==================================================================
Sam J. Marrocco
Sr. Visual Effects Artist/R&D
Travelling Pictures/GTN
Inferno, Flame, Houdini, Maya, All that cool stuff!
"The fact that no one understands you doesn't make you an artist."
==================================================================
caenogenesis
| I'm trying to get an application to run which is requiring more than
| 2gb of ram in a single process/task. Can anyone suggest any
| windows-based alternatives that would work?
|
| I've heard that Windows Server DataCenter can address more than 2gb
| per process, but can't--for the life of me--seem to locate anyone who
| knows anything in detail about it, or can sell it to me! MS's web
| site is all marketing, so it doesn't help me much.
|
| Any other 32-bit alternatives in windows I should be looking into?
Perhaps the people who wrote the application can help.
--
Gazwad
Freelance scientist and people tester.
Guardian: alt.os.windows-xp
Moderator: alt.warez.uk
http://angry.at/gazwad
http://gazwad.servebeer.com
Sam Marrocco
caenogenesis wrote:
> Sam Marrocco <samSPA...@twmi.rr.com> asked wistfully...
>
> Perhaps the people who wrote the application can help.
>
Unfortunately not.....we breaking new ground with this project.
caenogenesis
|
| Unfortunately not.....we breaking new ground with this project.
So you're writing the thing, and I was about to say "whatever fucking idiot
wrote the app is a moron".
Luckily I refrained, I had a suspicion it might be you.
relic
| I'm trying to get an application to run which is requiring more than
| 2gb of ram in a single process/task. Can anyone suggest any
| windows-based alternatives that would work?
|
| I've heard that Windows Server DataCenter can address more than 2gb
| per process, but can't--for the life of me--seem to locate anyone who
| knows anything in detail about it, or can sell it to me! MS's web
| site is all marketing, so it doesn't help me much.
|
| Any other 32-bit alternatives in windows I should be looking into?
Is 32GB enough?
http://www.microsoft.com/windows2000/advancedserver/evaluation/business/overview/scalable/default.asp
--
- relic -
Resident Psychic: alt.os.windows-xp
relic
BTW, think long and hard before considering the Data Center Edition... there's
some major differences between it and Server/Advanced Server Editions.
Sam Marrocco
caenogenesis wrote:
> Sam Marrocco <samSPA...@twmi.rr.com> asked wistfully...
>
> |
> | Unfortunately not.....we breaking new ground with this project.
>
> So you're writing the thing, and I was about to say "whatever fucking idiot
> wrote the app is a moron".
>
Your suspicion is incorrect. The application is Houdini, by Side Effects
Software, and would be anything from a casual rewrite/fix.
Sam Marrocco
relic wrote:
> | Is 32GB enough?
> |
> http://www.microsoft.com/windows2000/advancedserver/evaluation/business/overview/scalable/default.asp
>
> BTW, think long and hard before considering the Data Center Edition... there's
> some major differences between it and Server/Advanced Server Editions.
>
Yes it would be, if it worked. As stated, I have yet to speak to anyone
who can:
A-Tell me they've used Datacenter, and if it would work for me.
B-Can tell me where to buy it or how much it would cost.
Can you?
caenogenesis
|
| Your suspicion is incorrect. The application is Houdini, by Side
| Effects Software, and would be anything from a casual rewrite/fix.
So what platform are the people who are developing this package running it
on?
relic
|
||| Is 32GB enough?
|||
||
http://www.microsoft.com/windows2000/advancedserver/evaluation/business/overview/scalable/default.asp
||
|| BTW, think long and hard before considering the Data Center
|| Edition... there's
|| some major differences between it and Server/Advanced Server
|| Editions.
||
|
|
| Yes it would be, if it worked. As stated, I have yet to speak to
| anyone
| who can:
| A-Tell me they've used Datacenter, and if it would work for me.
| B-Can tell me where to buy it or how much it would cost.
|
| Can you?
Never used Datacenter, it kept being postponed for long enough for me to
retire (it was overkill anyway, but the "me too" forces wanted to implement
our Parallel Processing system on it). While I was still working, we used
Windows 2000 Advanced Server Edition with multiple 4 and 8 GB Nodes (it
supports up to 32GB if you can find a motherboard that will also) without
problems other than the early 2002 >64MB Registry problem (that has since been
fixed).
W2K Advanced Server runs ~$2700 (and up, depending on number of seats) here:
http://www.cdw.com/shop/products/default.asp?edc=215751
It will be replaced by Windows.NET sometime this year (so says Microsoft...
but .NET was supposed to be a mid-2002 product).
The only places I'm aware of that you can buy the W2K Datacenter version is
'with a system' from:
Bull
Compaq/HP
Dell
Fujitsu/Siemens
Hitachi
IBM
NEC
Unisys
Never heard of it being available (yet) as a stand-alone product for purchase.
Sam Marrocco
caenogenesis wrote:
> Sam Marrocco <samSPA...@twmi.rr.com> asked wistfully...
>
> |
> | Your suspicion is incorrect. The application is Houdini, by Side
> | Effects Software, and would be anything from a casual rewrite/fix.
>
> So what platform are the people who are developing this package running it
> on?
>
Standard win32/intel i86 32bit hardware, under windows pro,2000, nt and
linux.
We're the ones attempting to access such large amounts of data, hence
our attempts to learn about windows datacenter.
Sam Marrocco
relic wrote:
> ||
> || BTW, think long and hard before considering the Data Center
> || Edition... there's
> || some major differences between it and Server/Advanced Server
> || Editions.
> ||
> |
> |
> | Yes it would be, if it worked. As stated, I have yet to speak to
> | anyone
> | who can:
> | A-Tell me they've used Datacenter, and if it would work for me.
> | B-Can tell me where to buy it or how much it would cost.
> |
> | Can you?
>
> Never used Datacenter, it kept being postponed for long enough for me to
> retire (it was overkill anyway, but the "me too" forces wanted to implement
> our Parallel Processing system on it). While I was still working, we used
> Windows 2000 Advanced Server Edition with multiple 4 and 8 GB Nodes (it
> supports up to 32GB if you can find a motherboard that will also) without
> problems other than the early 2002 >64MB Registry problem (that has since been
> fixed).
>
If what you say is correct, then datacenter would be overkill for these
workstations, and advanced server might be the ticket.
What's meant by "multiple 4 and 8gb nodes"? Ideally, if I use a Tyan
dual xeon motherboard, I can get 4gb into 4 slots. Would Advanced server
edition then allow me to use *all* of that ram for a single process/task
(unlike xp pro, which limites processes/tasks to 2gb each)?
relic
|
| these workstations, and advanced server might be the ticket.
|
| What's meant by "multiple 4 and 8gb nodes"?
The system I worked with used up to 8 quad-XEON nodes connected by a
proprietary interconnect similar to a Banyon network. It was a Decision
Support Database engine that processed the database in parallel. Windows
restricted it to 8 nodes max, the UNIX versions scaled easily to 512 Nodes
(tested), and theoretically it should scale to 4096 nodes.
| Ideally, if I use a Tyan
| dual xeon motherboard, I can get 4gb into 4 slots. Would Advanced
| server edition then allow me to use *all* of that ram for a single
| process/task (unlike xp pro, which limites processes/tasks to 2gb
| each)?
Oh oh... That I don't know. Our application was pretty small, the remaining
memory was used as sorting cache and Fail-over redundancy. I don't know if
the W2K Server Editions have similar restrictions as XP, XP was built on W2K
Professional instead of the Server Editions. I searched around quickly in
Microsoft W2K Server and W2k Advanced Server documentation, but came up dry on
Process/Task size limitations. That would have to be asked directly to
Microsoft I'm afraid.
(BTW, there are W2K Server and W2K Advanced Server editions. Server edition
only runs on an SMP with up to 32-way CPUs, while Advanced Server runs on (up
to) 8-node MPP systems. But it only runs around $800 (it also supports up to
32GB).
Sam Marrocco
relic wrote:
> | Ideally, if I use a Tyan
> | dual xeon motherboard, I can get 4gb into 4 slots. Would Advanced
> | server edition then allow me to use *all* of that ram for a single
> | process/task (unlike xp pro, which limites processes/tasks to 2gb
> | each)?
>
> Oh oh... That I don't know. Our application was pretty small, the remaining
> memory was used as sorting cache and Fail-over redundancy.
Fortunately, the number of processors is less of an issue than the
amount of RAM we can address in a single process/task. I appreciate the
help and feedback you've provided....thanks much.
Any idea how I can find a Microsoft expert that can answer these type of
questions? I'll need to confirm the ram/process issue before I can go
much further.
Emailing MS about something like this is like showing up at
GeneralMotors front door and asking for lug nuts ;)
relic
|
| amount of RAM we can address in a single process/task. I appreciate
| the
| help and feedback you've provided....thanks much.
Any time.
|
| Any idea how I can find a Microsoft expert that can answer these type
| of questions? I'll need to confirm the ram/process issue before I can
| go
| much further.
|
| Emailing MS about something like this is like showing up at
| GeneralMotors front door and asking for lug nuts ;)
There's an Microsoft "Expert" permanently officed at my old company's
Development shop. Let me see if I can find out something on Monday. His
'area' is Enterprise development, so he /should/ know.
Your e-mail addy good without the spam stuff?
Sam Marrocco
> There's an Microsoft "Expert" permanently officed at my old company's
> Development shop. Let me see if I can find out something on Monday. His
> 'area' is Enterprise development, so he /should/ know.
>
Thanks, that would be great!
> Your e-mail addy good without the spam stuff?
Not sure what you mean...?
relic
|| Development shop. Let me see if I can find out something on Monday.
|| His 'area' is Enterprise development, so he /should/ know.
||
|
|
| Thanks, that would be great!
|
|| Your e-mail addy good without the spam stuff?
|
|
| Not sure what you mean...?
samSPA...@twmi.rr.com = samATtwmiDOTrrDOTcom
Sam Marrocco
relic wrote:
> I knew that someday Sam Marrocco would write:
> || There's an Microsoft "Expert" permanently officed at my old company's
> || Development shop. Let me see if I can find out something on Monday.
> || His 'area' is Enterprise development, so he /should/ know.
> ||
> |
> |
> | Thanks, that would be great!
> |
> || Your e-mail addy good without the spam stuff?
> |
> |
> | Not sure what you mean...?
>
> samSPA...@twmi.rr.com = samATtwmiDOTrrDOTcom
Ah....yes, that's correct. Sorry, I'm a little slow today....too much
coding is bogging my brains :)
relic
For applications that are I/O-intensive, such as database management systems
(DBMS), the use of a larger virtual address space can provide considerable
performance benefits.
Operating systems based on Microsoft(r) Windows NT(r) provide applications
with a 4 GB virtual address space. The virtual address space is divided such
that 2 GB is available to the application and the other 2 GB is available only
to the system. The 4GT RAM Tuning feature increases the memory available to
the application to 3 GB, and reduces the amount available to the system to 1
GB. This benefits applications that run on computers with more than 2 GB of
physical memory.
To enable this feature, add the /3GB switch to the Boot.ini file. This enables
applications to use the first 3 GB of the address space on the following
systems:
Windows .NET Server 2003 family
Windows XP Professional Edition
Windows 2000 Datacenter Server
Windows 2000 Advanced Server
Windows NT 4.0 Enterprise Edition
*Note:* Windows 2000 Server still have the 2 GB Limit!
To enable your application to use the 3 GB address space, set the
IMAGE_FILE_LARGE_ADDRESS_AWARE flag in the image header. The linker included
with Microsoft Visual C++ supports the /LARGEADDRESSAWARE switch to set this
flag. Setting this flag and then running the application on a system that does
not have 4GT support should not affect the application.
Use the following guidelines when enabling 4GT support within your
applications:
For a DLL that loads near the 2 GB boundary, there is a region of the 2 GB
space in which contiguous memory cannot be allocated using the VirtualAlloc
function.
Use the GlobalMemoryStatus function to retrieve the amount of total user
virtual space. Avoid using hard-wired constant definitions such as "#define
HIGHEST_USER_ADDRESS 0xC0000000". Try to detect the real value at runtime.
Avoid signed comparisons with pointers as they may cause applications to crash
on a 4GT-enabled system. A condition such as "if (pointer > 40000000)" is
false for a pointer that is above 2 GB.
Code that uses the highest bit to tag items (data value versus an address
value) will fail. For example, a 32-bit word might be considered a user- mode
address if below 0x80000000 and an error code if above. This is no longer true
with 4GT.
*AND:*
( Note: This still doesn't increase per process memory limit, but increases
Addressable memory space to more than 4 GB)
Physical Address Extension
The Physical Address Extension (PAE) enables applications to address more than
4 GB of physical memory. It is supported by Intel processors. The following
system can use PAE to take advantage of physical memory beyond 4 GB:
Windows .NET Enterprise Server
Windows .NET Advanced Server
Windows 2000 Datacenter Server
Windows 2000 Advanced Server
To enable PAE, you must use the /PAE switch in the Boot.ini file.
With PAE enabled, the operating system moves from two-level linear address
translation to three-level address translation. The extra layer of translation
provides access to physical memory beyond 4 GB. Instead of a linear address
being split into three separate fields for indexing into memory tables, it is
split into four separate fields; a 2-bit field, two 9-bit fields, and a 12-bit
field that corresponds to the page size implemented by Intel Architecture (4
KB).
Typically, a process can access up to 2 GB of memory address space (assuming
the /3GB switch was not used), with some of the memory being physical memory
and some being virtual memory. The memory manager uses PAE to provide more
physical memory to an application, which reduces the need to swap memory to
the page file and increases performance. The application itself is not aware
of the actual memory size. All of the memory management and allocation of the
PAE memory is handled by the memory manager independently of the application.
Applications that are 4GT-aware are likely to remain in physical memory rather
than be paged out, which increases their performance. The exception is when
the /3GB switch is used in conjunction with the /PAE switch. In this case, the
operating system does not use any memory in excess of 16 GB. Therefore, if the
system restarts with the /3GB entry in the Boot.ini file, and the system has
more than 16 GB of physical memory, the additional physical random access
memory (RAM) is not used by the operating system. Restarting the computer
without the /3GB switch enables the use of all of the physical memory.
Address Windowing Extensions (AWE) enables applications to address more than 4
GB. AWE enables an application to reserve physical memory as nonpaged memory,
then dynamically map portions of the nonpaged memory to its working set. This
enables memory-intensive programs to reserve large amounts of physical memory
for data without swapping to disk. Instead, the data is swapped between the
working set and reserved memory above the 4 GB range. The memory above 4 GB is
exposed to the memory manager and the AWE functions by PAE. Without PAE, AWE
is unable to reserve memory in excess of 4 GB.
Address Windowing Extensions
Address Windowing Extensions (AWE) is a set of extensions that allows an
application to quickly manipulate physical memory greater than 4GB. Certain
data-intensive applications, such as database management systems and
scientific and engineering software, need access to very large caches of data.
In the case of very large data sets, restricting the cache to fit within an
application's 2GB of user address space is a severe restriction. In these
situations, the cache is too small to properly support the application.
AWE solves this problem by allowing applications to directly address huge
amounts of memory while continuing to use 32-bit pointers. AWE allows
applications to have data caches larger than 4GB (where sufficient physical
memory is present). AWE uses physical nonpaged memory and window views of
various portions of this physical memory within a 32-bit virtual address
space.
AWE places a few restrictions on how this memory may be used, primarily
because these restrictions allow extremely fast mapping, remapping, and
freeing. Fast memory management is important for these potentially enormous
address spaces.
Virtual address ranges allocated for the AWE are not sharable with other
processes (and therefore not inheritable). In fact, two different AWE virtual
addresses within the same process are not allowed to map the same physical
page. These restrictions provide fast remapping and cleanup when memory is
freed.
The physical pages that can be allocated for an AWE region are limited by the
number of physical pages present in the machine, since this memory is never
paged - it is locked down until the application explicitly frees it or exits.
The physical pages allocated for a given process can be mapped into any AWE
virtual region within the same process. Applications that use AWE must be
careful not to take so much physical memory that they cause other applications
to page excessively or prevent creation of new processes or threads due to
lack of resources. Use the GlobalMemoryStatusEx function to monitor physical
memory use.
AWE virtual addresses are always read/write and cannot be protected via calls
to
Sam Marrocco
relic wrote:
> *4GT RAM Tuning*
>
Thanks for the info....I'm going to begin experimenting with this tomorrow.
relic
|| *4GT RAM Tuning*
||
|
|
| Thanks for the info....I'm going to begin experimenting with this
| tomorrow.
Any time.
AND, it's in XP Pro so you don't need to buy anything new right now.