Showing posts with label performance. Show all posts
Showing posts with label performance. Show all posts

"Fixed" .NET Framework Updates Still Cause High CPU on AAD Connect Servers

Friday, July 20, 2018
AKA - "What's that burning smell?" or "Why does my server sound like a Boeing 747?"

Continuing my rant about buggy Windows Updates -- especially .NET Framework -- I've found that even the latest "fixed" .NET Framework updates still cause high CPU on AAD Connect servers.


On Windows Server 2012 R2 the offending update is 2018-07 Security and Quality Rollup for .NET Framework 3.5, 4.5.2, 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2 for Windows 8.1 and Server 2012 R2 for x64 (KB4340558). This "quality" rollup is actually a package that contains three different .NET Framework updates:
  • KB4338415 - Security and Quality Rollup updates for .NET Framework
  • KB4338419 - Security and Quality Rollup updates for .NET Framework
  • KB4338424 - A rollup update that is not available as a stand-alone package
It turns out that KB4338419 is the real offender that causes a race condition on the Microsoft.Identity.Health.AadSync.MonitoringAgent.Startup.exe process. If your AAD Connect server is experiencing high CPU for this process, uninstall KB4338419 or the entire KB4340558 package.

Be advised:
  • There are different KB packages for different OS's. Find the update that matches your OS in "installed updates"
  • Uninstalling these updates requires a restart (again)
  • .NET will recompile its assemblies when the computer restarts (again)
  • Windows Update will recommend installing these updates again. Do not install them. Better yet, hide them so Windows Update no longer offers them.

Read more ...

.WHY?

Tuesday, July 17, 2018
I'm growing increasingly frustrated with .NET Framework updates. Microsoft released .NET Framework 4.7.2 as an "important" update on July 10, 2018 and problem reports immediately started pouring in.

I reported earlier that .NET Framework 4.7.2 is not supported on any Exchange Servers and many users, including myself, were seeing high CPU usage on AAD Connect servers.



Doesn't anyone test this stuff, or is that our job as customers? What's the point of pushing out an update as "important" that core functionality doesn't support and actually harms the infrastructure that enterprises rely on? It takes a server restart to install this update and another one to remove it. And every time we get a .NET Framework update and reboot, server performance is affected while all the .NET Framework assemblies are recompiled. NO MORE.

From now on, I don't plan to install any .NET Framework updates unless,
  1. It's required for server functionality.
  2. It fixes a security vulnerability that actually affects me.
  3. It's been deployed worldwide for at least two weeks.
Not everyone is a developer who has to have the latest API updates. 

Microsoft, stop pushing buggy updates as "important". It's embarrassing. Get your quality issues sorted out, test, and coordinate with other product teams.

Read more ...

Fix for Performance Counter Updating Error (Event 106) on Exchange Servers

Tuesday, November 29, 2016
You may see multiple warnings or errors in the Application event log on Exchange servers for event 106, indicating that performance counters could not be updated.

Multiple Event ID 106 - Perf Counter Events
An example event 106 reads,
Log Name:      Application
Source:        MSExchange Common
Date:          11/29/2016 12:24:28 PM
Event ID:      106
Task Category: General
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      EX16B.contoso.com
Description:
Performance counter updating error. Counter name is Percentage of Failed Offline GLS Requests in Last Minute, category name is MSExchange Global Locator OfflineGLS Processes. Optional code: 3. Exception: System.InvalidOperationException: The requested Performance Counter is not a custom counter, it has to be initialized as ReadOnly.
   at System.Diagnostics.PerformanceCounter.InitializeImpl()
   at System.Diagnostics.PerformanceCounter.get_RawValue()
   at Microsoft.Exchange.Diagnostics.ExPerformanceCounter.set_RawValue(Int64 value)
The TechNet article, "'Performance counter updating error' after you install an Exchange Server 2013 cumulative update", indicates this is a problem on Exchange 2013 after applying a CU, but I've seen this happen after applying an update rollup, cumulative update, or .NET Framework update on Exchange 2010, Exchange 2013, or Exchange 2016 servers.

I wrote the following PowerShell script to reload the Exchange performance counters. You run it directly on the affected Exchange server from an elevated PowerShell or EMS prompt. The script will reload all the performance counters in the %ExchangeInstallPath%\setup\perf directory.
Add-PSSnapin Microsoft.Exchange.Management.PowerShell.Setup
$perfcounters = Get-ChildItem "$env:ExchangeInstallPath\Setup\Perf\" *.xml | Where-Object {!($_.psiscontainer)}
$perfcount = $perfcounters.Count

foreach ($perfcounter in $perfcounters) {
New-PerfCounters -DefinitionFileName $perfcounter.FullName -ErrorAction SilentlyContinue
Write-Progress -Activity "Reloading $perfcount Exchange Performance Counters" -PercentComplete (($i++ / $perfcount) * 100)}
The script may take a few minutes to run and will reload all the Exchange performance counters. Once complete, the event 106 warnings or errors should stop.


Read more ...

File-level Defragmentation of Exchange 2013 Volumes

Friday, July 26, 2013
By default, Windows Server 2012 automatically runs a weekly drive optimization (file-level defragmentation) on all drives. It also automatically optimizes new drives when they are added.

Now that you can install Exchange 2013 on Windows Server 2012 should you disable this feature as part of your server builds?

The latest information I could find from Microsoft on this subject is from Nino Bilic (2004): Do we need to file-level defragment Exchange database drives? and Mike Lagase (2011): How fragmentation on incorrectly formatted NTFS volumes affects Exchange.

Nino writes,
"The bottom line really is - you do not HAVE to file-level defrag the Exchange database drives. Exchange reads and writes to it's databases in very random fashion. Large sequential reads and writes will see much more improvement from file system defrag than Exchange databases will. But if you really WANT to do it - I would do it the old-fashioned way: move the databases off to some other volume, file system defrag the drive and then move the databases back... Or at least make sure you have a good backup, dismount the databases and file-system defrag them.
Mike writes,
"Note that it is still not recommended to run disk defragmentation software on Exchange server volumes, but there are times where file level fragmentation can cause significant performance problems on a server merely by the way data is being written to the disk. If optimal and/or recommended settings are not used when creating the volumes, this file fragmentation issue can occur much quicker. The majority of Exchange files are in use so running any regular disk defragmentation programs on the server will not help with this situation. If necessary, the only way to resolve this is to take all Exchange resources offline to ensure none of the files are in use and then defragment the disk to make the files contiguous on the disk once again."
That's Mike's emphasis, not mine.

So, while it's not necessarily going to break anything if you defragment your disks, it's best practice NOT to do it.  Especially letting Windows Server 2012 do it automatically every Wednesday at midnight, as is the default.  Also consider other workloads that may be running at the same time, such as Exchange backups.

Note that this recommendation normally applies to all local disks on an Exchange server, not just the drives that contain mailbox databases and transaction logs.  That's because the SMTP queue and temporary files are written to the same drive where the Exchange 2013 binaries are installed, usually the C: drive unless you've manually moved them to a different drive.

The Exchange product group will be officially adding this guidance for Exchange 2013 server builds on Windows Server 2012 in the near future.

Here's how to turn automatic drive optimization off in Windows Server 2012:

  • Using File Explorer, view the properties of the C: drive or any other local drive.
  • On the Tools tab, click Optimize.
  • You will see that all local drives are being optimized automatically every week:
  • Click the Change Settings button.
  • Clear the check box to Run on a schedule, as shown below:
  • Click OK, Close, and OK to save the settings.
Disabling drive optimization in this way will disable optimization for all drives.  It does not disable your ability to manually run a disk optimization on any drive.
Read more ...

Western Digital Green vs Black Drive Comparison

Saturday, February 18, 2012
In a recent post I described my new blistering fast Windows 8 Server, which includes a parts list.  This server features a 120GB SDD SATA III 6.0Gb/s drive for the operating system and uses a single 2TB Western Digital Green SATA III 6.0Gb/s drive (WD20EARX-00PASB0) for VM and data storage.

It has been suggested by some of my readers that the WDC Green drive will not provide suitable performance compared to a WDC Black SATA III drive.  They also wondered what the true power savings is between the Green and the Black drive.  The Green drive uses less power by spinning at slower RPMs (variable ~5400 RPM vs 7200 RPM for the Black).

I decided to purchase a Western Digital Caviar Black SATA III 6.0Gb/s drive (WD2002FAEX-007BA) to run benchmarks against and compare the two drives side-by-side using HD Tune Pro 5.00 and Microsoft Exchange Server Jetstress 2010 (64 bit).

I ran each set of tests for the Green drive, then replaced it with the Black drive and ran the same set of tests on my new server.  I also ran the the tests while the server was plugged into a P3 Kill A Watt Electricity Load Meter and Monitor to accurately measure power consumption by the kilowatt-hour for comparison.

HD Tune Pro Benchmarks
The following are the benchmark test results for both drives.  The Green drive is on the left and the Black is on the right.

Benchmark Results
The Black drive delivers 17.9% better average transfer speed.  The access time was 17.6ms for the Green vs. 12.0ms for the Black.  I was surprised to see that CPU usage was much higher on the Green (6.0%) vs the Black (2.4%).


File Benchmark Results
The File Benchmark test measures read/write transfer speed using a 500MB file in 4KB blocks.  The Black drive achieved 11.5% better performance using 4KB sequential access and 28.2% better using 4KB random access.


Random Access Results
The Random Access test measures the performance of random read or write operations with varying data sizes (512 bytes - 1MB).  Again, the Black drive performed better across the board with an average 31.2% improved performance.  It also offers much better access times.

It's notable that the Green drive performed this test nearly silently, while the Black drive sounded like a Geiger Counter at Fukushima.  Neither of these drives feature AAM (Automatic Acoustic Management) so this does not impact the results (and cannot be adjusted).


Other Test Results
This benchmark runs a variety of tests which determine the most important performance parameters of the hard drive.  The Black drive offers 35.3% better random seek and 18.3% better sequential read performance.  It also has better transfer speeds from its cache.  Both drives feature a 64MB cache.



Exchange JetStress
I ran Exchange 2010 JetStress on each drive to get an accurate IOPS profile for Exchange 2010 SP2 use.  JetStress was configured for a two-hour test using a single 1TB database and one thread.

  • The Green drive achieved 47.396 IOPS with 10.751ms latency.
  • The Black drive achieved 64.57 IOPS with 15.180 latency.

I'm not sure why the Black drive's latency was higher than the Green, given the benchmark tests above, but I ran that test twice and got the same results each time.  Even so, the Black drive delivered 26.6% more IOPS.



Power Analysis
Green Drive1.10 KW at 27.5 hours
Energy use per hour = (1.1 KWH)/(27.5 hours) = 0.04 KWH per hour of use
Energy use per day = (0.04 KWH/hour)(24 hours/day) = 0.96 KWH over a full day
Cost per day = (0.96 KWH)(18.5 cents/KWH) =  17.8 cents per day

Energy use per year = (0.96 KWH/day)(365 days/year) = 350 KWH/year
Cost per year = (350 KWH/year)(18.5 cents/KWH) = $64.82 per year.

350 KWH = ~700 lbs of greenhouse gas to the atmosphere per year.

Black Drive0.72 at 14.75 hours
Energy use per hour = (0.72 KWH)/(14.75 hours) = 0.049 KWH per hour of use
Energy use per day = (0.049 KWH/hour)(24 hours/day) = 1.18 KWH over a full day
Cost per day = (1.18 KWH)(18.5 cents/KWH) =  21.83 cents per day

Energy use per year = (1.18 KWH/day)(365 days/year) = 431 KWH/year
Cost per year = (431 KWH/year)(18.5 cents/KWH) = $79.74 per year.

431 KWH = ~860 lbs of greenhouse gas to the atmosphere per year.

Result: The WDC Green drive uses 18.8% less energy than the Black drive.



Conclusion
It's obvious from the test results above that the Western Digital Caviar Black drive performs better than the Green drive.  At the time of this writing the Green drive costs $139 and the Black is $249.  That's a 44% premium for a drive that performs on average 24% better.

In real-life observations I don't really see that much difference in performance between the two at this time.  However, this Hyper-V server has twice as much RAM as my last server so it will potentially be hosting many more VMs (and will have a higher IO load).  For this reason I decided to keep the Black drive, even though it costs more, it's a bit noisier when it's working hard and uses more energy.  I like muscle cars, too.  :)

If you plan to do RAID, I would most definitely recommend the Black drive because it spins at a consistent 7200 RPM.  Reports say that the variable RPMs on the Green drive can cause read/write errors.

I hope you find this information useful.

Read more ...

How To Enable Change Notification On All Site Links

Wednesday, March 18, 2009

Normally, there are two replication intervals for Active Directory in a Windows domain: Intra-site (replication between DCs in the same site) and Inter-site (replication between DCs in different Active Directory sites).
Intra-site replication is very fast - typically around 15 seconds. This schedule can be configured via the registry using the following values in the HKLM\SYSTEM\CurrentControlSet\Service\NTDS\Parameters key:

Replicator notify pause after modify (secs)It is a REG_DWORD value of 15 by default
Replicator notify pause between DSAs (secs)It is a REG_DWORD value of 3 by default
See Microsoft TechNet (Active Directory Replication Tools and Settings) for a thorough explanation of what these keys do.Inter-site replication is dictated by the schedule associated with the replication connection in Active Directory Sites and Services. Using this GUI you can specify that the connector never replicates or to replicate once, twice or four times per hour.
Note: The inter-site replication schedule runs based on the server startup time. For example, if the DC starts up at 12:10pm and the replication connector's schedule is set to twice per hour, replication on this connector will occur at 12:10pm, 12:40pm, etc.
But what if you want Intersite replication to occur more frequently than every 15 minutes? For this, you must enable Change Notification on the Active Directory site link. How you do this depends on which OS is on your DC.

For Windows 2003 Domain Controllers:
  • Open ADSIEdit.msc (in the Windows Support Tools) as a Domain Admin
  • Open the Configuration naming context
  • Navigate to Sites > Inter-Site Transports > IP
  • Right-click the siteLink to modify in the results pane and click Properties
  • Locate the options attribute and edit the value from 0 to 1
  • Click OK and repeat for other siteLinks, as necessary.

For Windows 2008 and Windows 2008 R2 Domain Controllers:
You can use the same method as Windows Server 2003 DCs or you can edit the values directly from AD Sites and Services, as follows.
  • Locate the Inter-Site Transport site link to modify, typically DEFAULTIPSITELINK, in AD Sites and Services
  • Right-click the site link and choose Properties
  • Click the Attribute Editor tab
  • Locate the options attribute and edit the value from 0 to 1
  • Click OK and repeat for other Site Links, as necessary.
I also wrote two VBScripts for displaying and configuring Change Notification:
  • DisplayChangeNotification.vbs displays the current value of the options attribute on each site link in the Active Directory domain where it is run.
  • EnableChangeNotification.vbs will enable Change Notification on all site links in the Active Directory domain where it is run by changing the options value to 1.
Both scripts are in the ChangeNotificationScripts.zip file, located here.
Read more ...

How to Move the SMTP Queue in Exchange 2007

Monday, January 12, 2009

Unlike previous versions of Exchange, all SMTP queue activity in Exchange Server 2007 happens in a new ESE database.

By default, this database (and its logs) exists in the C:\Program Files\Microsoft\Exchange Server\TransportRoles\data\Queue folder. You may wish to move this database and its logs to a seperate physical volume for better performance. Here's how to do this:

To Change the Database Path:

1. Open the EdgeTransport.exe.config file in the C:\Program Files\Microsoft\Exchange Server\Bin folder using Notepad

2. Edit the value of the line containing add key="QueueDatabasePath" to reflect the new path. For example:

add key="QueueDatabasePath" value="D:\QueueDB"

To Change the Database Logs Path:

3. Edit the value of the line containing add key="QueueDatabaseLoggingPath" to reflect the new path. For example:

add key="QueueDatabaseLoggingPath" value="D:\QueueLogs"

4. Save the file and restart the Microsoft Exchange Transport service

Read more ...

Your Troubleshooting PAL

Wednesday, July 23, 2008

How many times have you been faced with a performance issue with a computer and you don't really know where to start? Sure, you can fire up Performance Monitor (perfmon) and start collecting data for analysis, but which counters do you collect and how do you identify a bottleneck?

Perfmon can gather tons of information and pouring over all that data for analysis can be a daunting task. Enter Performance Analysis of Logs (PAL), a new and powerful tool that reads in a performance monitor counter log in any known format and analyzes it using complex, but known thresholds. The tool produces an HTML report which reports important performance counters and displays alerts when thresholds are exceeded.

PAL is a free open source application developed by Microsoft and is hosted on CodePlex, Microsoft's open source project hosting web site. It requires two other free pieces of software on the computer where PAL will run:

Log Parser 2.2
Log parser is a powerful, versatile tool that provides universal query access to text-based data such as log files, XML files and CSV files, as well as key data sources on the Windows operating system such as the Event Log, the Registry, the file system, and Active Directory. PAL uses the Log Parser tool to query perform logs and to create charts and graphs for the PAL report.

Microsoft Office Web Components 2003
Log Parser requires the Office Web Components 2003 in order to create charts.

Note: Because there is no 64-bit version of the Microsoft Office Web Components, PAL only runs on x86 platform computers.

To use PAL, you begin by collecting performance data from the target machine using perfmon. Typically, I collect the Memory, Network Interface, Physical Disk, Processor and System counters to begin with. Once you've collected some data run PAL and walk through the wizard. Be sure to answer the Question Variable Names at the bottom of the Threshold File page. The variables are Number of Processors, use of the /3GB switch, is the target a 64-bit computer, total RAM and whether it has a kernel dump configured. Step through the rest of the wizard and PAL will create a batch file, run it and display the output as a graphical report in your web browser. Very cool!!!

You can view a LiveMeeting streaming video training of PAL here.

Read more ...

Have you examined your Pagefile lately?

Tuesday, September 18, 2007
Whenever you build a new server or workstation Windows automatically sets the pagefile size based on the amount of RAM available. Usually, it sets the minimum amount of RAM to 1.5X RAM and the maximum to 2X RAM. So, for a server with 2GB RAM the pagefile minimum might be 3072MB and the maximum size might be 4096MB.

That might work well for your base build, but what about after you add the applications you built the server to do? The pagefile requirements change dramatically after you install SQL, Exchange, antivirus, etc.

You can view the pagefile settings by checking your system settings (Right-click Computer Properties Advanced tab Performance settings Advanced Change). Take a look at the Recommended setting, as in the example in this article.

I recommend setting the Initial size to the Recommended size (4509MB in the example above), and the Maximum size to 2x RAM (8096MB for this server). This will have a truly awesome affect on server performance! I especially find big performance gains in Exchange and SQL.

It's interesting to note that even though Windows recommends 4509MB for a pagefile, the system's currently allocated pagefile size is only the inital size. You would think that Windows would adjust the pagefile up to the maximum size of 4096MB, but it doesn't.

I recommend taking a look at your pagefile settings after installing your applications and occasionally (maybe every 6 months) while in production and adjust as needed.
Read more ...