Low Virtual Memory Conditions in Windows 2012R2 and Exchange Server 2013 CU23

[selahattin şadoğlu]

Hi,

The server is a virtualized Server 2012R2, running on a Server 2012R2 host. The virtual server has been set with 20 vCPU's and 40GB RAM. We have about 3200 mailboxes spread between 12 databases (500mb/1Gb/3Gb/Archive)

We have 2 MBX and 2 CAS machines.

Is anyone aware of anything that could cause this issue? memory leak ?

EVENT IDs on DB01 machine:

1 -

Process 'msexchangerepl', application 'msexchangerepl.exe' had the following resource usage characteristics during the last global activity rollup cycle. The duration of the last global activity rollup cycle was '06:00:00'.

I32:ATE.C[UNINSTR]=7444;F:ATE.AL[UNINSTR]=0.2677324;I32:ADS.C[UNINSTR]=7490;F:ADS.AL[UNINSTR]=1.466356;I32:ADR.C[UNINSTR]=4;F:ADR.AL[UNINSTR]=2.268075

2 -

The indexing of mailbox database DATABASE01 encountered an unexpected exception. Error details: Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. ---> Microsoft.Exchange.Search.Engine.FeedingSkippedForCorruptionException: "Feeding was skipped for '768f42ee-4c4d-4bc0-9ac8-23fc8f136add (DATABASE01)' due to the state 'Failed', error code: 'CatalogCorruption', failure code: '2400519', failure reason: 'Failed to initialize FastServer: Unable to init fastserver'."

at Microsoft.Exchange.Search.Engine.SearchFeedingController.InternalExecutionStart()

at Microsoft.Exchange.Search.Core.Common.Executable.InternalExecutionStart(Object state)

--- End of inner exception stack trace ---

at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)

at Microsoft.Exchange.Search.Engine.SearchRootController.ExecuteComplete(IAsyncResult asyncResult)

3 -

Log Name: System

Source: Microsoft-Windows-Resource-Exhaustion-Detector

Date: 6/30/2022 5:21:49 PM

Event ID: 2004

Task Category: Resource Exhaustion Diagnosis Events

Level: Warning

Keywords: Events related to exhaustion of system commit limit (virtual memory).

User: SYSTEM

Computer: exch1.contoso.local

Description:

Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: noderunner.exe (29376) consumed 1057222656 bytes, noderunner.exe (33376) consumed 1013346304 bytes, and EdgeTransport.exe (37792) consumed 983609344 bytes.

Event Xml:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

<System>

<Provider Name="Microsoft-Windows-Resource-Exhaustion-Detector" Guid="{9988748E-C2E8-4054-85F6-0C3E1CAD2470}" />

<EventID>2004</EventID>

<Version>0</Version>

<Level>3</Level>

<Task>3</Task>

<Opcode>33</Opcode>

<Keywords>0x8000000020000000</Keywords>

<TimeCreated SystemTime="2022-06-30T14:21:49.988850800Z" />

<EventRecordID>3954135</EventRecordID>

<Correlation ActivityID="{4FEBAF5C-4059-40D0-8FDA-949D796E31CE}" />

<Execution ProcessID="2796" ThreadID="2856" />

<Channel>System</Channel>

<Computer>exch1.contoso.local</Computer>

<Security UserID="S-1-5-18" />

</System>

<UserData>

<MemoryExhaustionInfo xmlns="http://www.microsoft.com/Windows/Resource/Exhaustion/Detector/Events">

<SystemInfo>

<SystemCommitLimit>177891254272</SystemCommitLimit>

<SystemCommitCharge>177622863872</SystemCommitCharge>

<ProcessCommitCharge>21517692928</ProcessCommitCharge>

<PagedPoolUsage>152884748288</PagedPoolUsage>

<PhysicalMemorySize>42949136384</PhysicalMemorySize>

<PhysicalMemoryUsage>39846916096</PhysicalMemoryUsage>

<NonPagedPoolUsage>677617664</NonPagedPoolUsage>

<Processes>230</Processes>

</SystemInfo>

<ProcessInfo>

<Process_1>

<Name>noderunner.exe</Name>

<ID>29376</ID>

<CreationTime>2022-06-30T12:03:29.476843800Z</CreationTime>

<CommitCharge>1057222656</CommitCharge>

<HandleCount>2363</HandleCount>

<Version>16.0.1497.0</Version>

<TypeInfo>65</TypeInfo>

</Process_1>

<Process_2>

<Name>noderunner.exe</Name>

<ID>33376</ID>

<CreationTime>2022-06-30T14:13:46.509001500Z</CreationTime>

<CommitCharge>1013346304</CommitCharge>

<HandleCount>2790</HandleCount>

<Version>16.0.1497.0</Version>

<TypeInfo>66</TypeInfo>

</Process_2>

<Process_3>

<Name>EdgeTransport.exe</Name>

<ID>37792</ID>

<CreationTime>2022-06-30T13:24:17.455846300Z</CreationTime>

<CommitCharge>983609344</CommitCharge>

<HandleCount>2983</HandleCount>

<Version>15.0.1497.26</Version>

<TypeInfo>67</TypeInfo>

</Process_3>

<Process_4>

<Name>mmc.exe</Name>

<ID>21540</ID>

<CreationTime>2022-06-30T13:14:19.673328600Z</CreationTime>

<CommitCharge>78307328</CommitCharge>

<HandleCount>417</HandleCount>

<Version>6.3.9600.18910</Version>

<TypeInfo>136</TypeInfo>

</Process_4>

<Process_5>

<Name>explorer.exe</Name>

<ID>13048</ID>

<CreationTime>2022-06-30T09:03:42.700967100Z</CreationTime>

<CommitCharge>54497280</CommitCharge>

<HandleCount>1240</HandleCount>

<Version>6.3.9600.18231</Version>

<TypeInfo>144</TypeInfo>

</Process_5>

<Process_6>

<Name>Taskmgr.exe</Name>

<ID>13084</ID>

<CreationTime>2022-06-30T10:22:03.885448900Z</CreationTime>

<CommitCharge>19111936</CommitCharge>

<HandleCount>347</HandleCount>

<Version>6.3.9600.17415</Version>

<TypeInfo>152</TypeInfo>

</Process_6>

</ProcessInfo>

<PagedPoolInfo>

<Tag_1>

<Name>NfRE</Name>

<PoolUsed>143099143104</PoolUsed>

</Tag_1>

<Tag_2>

<Name>MmSt</Name>

<PoolUsed>234191888</PoolUsed>

</Tag_2>

<Tag_3>

<Name>Toke</Name>

<PoolUsed>163057552</PoolUsed>

</Tag_3>

</PagedPoolInfo>

<NonPagedPoolInfo>

<Tag_1>

<Name>EtwB</Name>

<PoolUsed>276729888</PoolUsed>

</Tag_1>

<Tag_2>

<Name>ConT</Name>

<PoolUsed>43421696</PoolUsed>

</Tag_2>

<Tag_3>

<Name>TBIM</Name>

<PoolUsed>22681744</PoolUsed>

</Tag_3>

</NonPagedPoolInfo>

<ExhaustionEventInfo>

<Time>2022-06-30T14:21:42.775832400Z</Time>

</ExhaustionEventInfo>

</MemoryExhaustionInfo>

</UserData>

</Event>

4 -

Information Store - db22 (4128) db22: The system is experiencing memory allocation failures that are preventing proper operation of database 'db22' ('408ee778-19d8-4a6c-8fe7-03c6ee856df6'). The error could occur due to a mis-configuration or excessive memory consumption on the machine.

https://imgur.com/a/FaoiHmm

esx host : https://imgur.com/a/e5yCE9M

https://ark.intel.com/content/www/us/en/ark/products/199345/intel-xeon-gold-6238r-processor-38-5m-cache-2-20-ghz.html

[Michael B. Smith]

What is swap set to?

 

Memory is a little light, but this value is ridiculous: <SystemCommitCharge>177622863872</SystemCommitCharge>

--
You received this message because you are subscribed to the Google Groups "ntsysadmin" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ntsysadmin+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/9a779b12-6450-4e6b-8b6c-2979be206594n%40googlegroups.com.

[selahattin şadoğlu]

SWAP config :  https://imgur.com/a/0cEkMru  

<SystemCommitCharge>177622863872</SystemCommitCharge>  what does mean by this? 177622863872 equal  177GB? Solution :  add more RAM ? 

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/6dbd0cf6405844ffb9c172cc771beb9d%40smithcons.com.

--

System Administrator

Selahattin ŞADOĞLU

[brett...@hotmail.com]

The telling event for me is this:

Log Name: System

Source: Microsoft-Windows-Resource-Exhaustion-Detector

Date: 6/30/2022 5:21:49 PM

Event ID: 2004

Task Category: Resource Exhaustion Diagnosis Events

Level: Warning

Keywords: Events related to exhaustion of system commit limit (virtual memory).

User: SYSTEM

Computer: exch1.contoso.local

Description:

Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: noderunner.exe (29376) consumed 1057222656 bytes, noderunner.exe (33376) consumed 1013346304 bytes, and EdgeTransport.exe (37792) consumed 983609344 bytes.

What the resource governor is doing there is saying it found some large memory consuming processes, and then likely asked them to shed memory or be killed.

The upshot is, either it's time to add more memory to the VMs, or time to restart the OS. If your situation hasn't changed much (same # databases and users, no real load change) and this is only now just coming up, I'd just restart it and watch in the future. You're not going to get any attention on this code base even if it was a "leak", which I wouldn't typify it as... 

My more obvious process that behaves less than ideally is IIS on my Exch2016 environment, and I see memory usage creep up over time and if we are taking the system out for maintenance then the usage drops way back down again. Over a month or so it grows and peaks, but about then we're susually into another maintenance cycle so it never becomes an operational issue.

[Philip Elder]

That looks really lopsided to me as far as RAM to vCPU ratio. What’s the host processor and RAM setup?

 

40GB with that kind of business is really short in my mind especially given the vCPU count.

 

Philip Elder MCTS

Senior Technical Architect

Microsoft High Availability MVP

E-mail: Phili...@mpecsinc.ca

Phone: +1 (780) 458-2028

Web: www.mpecsinc.com

Blog: blog.mpecsinc.com

Twitter: Twitter.com/MPECSInc

Skype: MPECSInc.

 

Please note: Although we may sometimes respond to email, text and phone calls instantly at all hours of the day, our regular business hours are 8:00 AM - 5:00 PM, Monday thru Friday.

 

From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> On Behalf Of selahattin sadoglu
Sent: Sunday, July 3, 2022 09:34
To: ntsysadmin <ntsys...@googlegroups.com>
Subject: [ntsysadmin] Low Virtual Memory Conditions in Windows 2012R2 and Exchange Server 2013 CU23

 

Hi,

--

[selahattin şadoğlu]

Hi,

According to official document, 

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/solutions/vmware-microsoft-exchange-2013-best-practices-guide.pdf

Virtual machines, including those running Exchange 2013, should be configured with multiple virtual
sockets/CPUs. Only use the Cores per Socket option if the guest operating system requires the change
to make all vCPUs visible. Virtual machines using vNUMA might benefit from this option, but the
recommendation for these virtual machines is generally to use virtual sockets (CPUs in the web client).
Exchange 2013 is not a NUMA-aware application, and performance tests have not shown any significant
performance improvements by enabling vNUMA.

ESX Host:

https://imgur.com/a/pgGSqWM

VM :

https://imgur.com/a/YaCaMLa

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/91f5e906fb0742aaa0b309f07ff32959%40MPECSInc.Ca.

[brett...@hotmail.com]

yeah but what is the MEMORY configuration ??

[Philip Elder]

That looks wrong to me.

 

The physical server has _two_ sockets with 28 cores _per socket_.

 

For the VM we would set up 2 sockets with 10 cores per socket. Follow the _physical_ layout with the virtual layout keeping in mind how the CPU scheduler works at the hardware level.

 

How much memory per socket/CPU? That’s the next question.

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/CAKXd-rAwSoGBQJgLr%2BO8cShJpWWvsLUGvk%2BrFGJ%3DaSKCzJ_TpQ%40mail.gmail.com.

[Michael B. Smith]

You are outside proper bounds for Exchange 2013.

 

https://docs.microsoft.com/en-us/exchange/exchange-2013-system-requirements-exchange-2013-help

 

See the pagefile section.

 

Without running a mailbox calculator for your environment (which would require a bunch more information), I can’t be sure, but you probably need 64 GB RAM.

 

Philip is correct that you need to be aware of potential NUMA issues.

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/CAKXd-rCoSb%3DNQJPkEaguULg%2BXp588mXc_Czn-6c1aZxZJ1FvNQ%40mail.gmail.com.

[selahattin şadoğlu]

Hi

https://imgur.com/a/J5YmCQD

According to document , it saying

NOTE: The 2-socket prescription is based on Microsoft’s restated requirements for a single Exchange
Server.
While VMs using vNUMA may benefit from this option, the recommendation for these VMs is to use virtual
sockets (CPUs in the web client). Exchange Server 2019 is not a NUMA-aware application and
performance tests have shown no significant performance improvements by enabling vNUMA. However,
Windows Server 2019 OS is NUMA-aware and Exchange Server 2019 (as an application) does not
experience any performance, reliability, or stability issues attributable to vNUMA.

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/solutions/business-critical-apps/exchange/vmw-microsoft-exchange-server-2019-on-vmware-best-practices.pdf

NUMA :

https://imgur.com/a/nn7IdP9

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/76669397ee544032b29b8c0b10bca8f9%40MPECSInc.Ca.

[Philip Elder]

I read that as 1.5TB per node then?

 

If yes, then the NUMA boundary on one node will be 768MB which is what is installed per CPU in the system.

 

With that we’d set up as follows:

• QTY 1 Socket
• QTY 16 vCPUs
• QTY 64GB vRAM

 

Exchange likes memory lots of it. At the first MEC (Microsoft Exchange Conference) I remember the team extolling the fact that their dogfood was on a single SATA spindle for four hundred (400) busy mailboxes.

 

A VM should never swap. That’s certain death for its livelihood which is the workload(s) running on it.

 

With the NUMA barrier so high we don’t need to be too concerned about the CPU scheduler flinging bits between the two CPUs which is costly for performance.

 

By limiting the VM to _one_ socket we eliminate the performance cost of having two sockets involved.

 

Please get the virtual architecture in line with the physical architecture.

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/CAKXd-rBTwvJfS%2BRMFFQA46Zf5nsN4F7%3D0Jne%2B9Aryu_shq6nBQ%40mail.gmail.com.