How to Use a Recovery Database in Exchange 2010

This is another in my series of articles on Exchange 2010.  In this post I'll be writing about the Recovery Database feature in Exchange 2010.

Exchange 2010 no longer has the notion of Storage Groups, which were used in Exchange 2007 and 2003 to contain logical groupings of databases.  E2010 now simply lets you create databases on mailbox servers.  E2010 Standard Edition lets you create up to 5 databases per server. The Enterprise Edtion scales up to 100 databases per server.

In Exchange 2003/2007 you could restore a database "on top" of an original database to replace the existing database, or you could restore the database "along side" the existing database to recover select mailboxes or items.  You can do the same thing with Exchange 2010.  The difference is that in Exchange 2003/2007, you created a Recover Storage Group (RSG) to restore the database into.  In Exchange 2010, you simply restore the database and connect to it as a Recovery Database (RDB).  Here's how you do it in Exchange 2010.

Note: Ross Smith IV has a great article on single item recovery in Exchange 2010.  This assumes that the item can be recovered from the dumpster.  This article covers how to restore from a backup when the item cannot be recovered from the dumpster.  For example, on the rare occasion when a user realizes he/she deleted a folder or item past the dumpster retention period.

First, you have to have a good backup that contains the item to be recovered.  Windows Server 2008 and Windows Server 2008 R2 have the built-in Windows Server Backup feature.  I cover how to use WSB to backup Exchange here.

Now you must restore the data, but redirect it to another location.  In Windows Server Backup, this is done by choosing to recover the Exchange application (detailed in my previous article) and recovered to another location.  Typically, this is a new folder on the same Exchange server:

Once the recovery is complete, the database (EDB file) and transaction logs (LOG files) will reside in the new recovery D:\Recovery folder.  Note that WSB will not create this folder, it must already exist.

Now you need to add this database to the Exchange mailbox server as a Recovery Database. Currently, this is done using the Exchange Management Shell (EMS), as there is no way to do this from the GUI.  Run the following command to create a Recovery Database:

New-MailboxDatabase -Recovery -Name RDB1 -Server EX1 -EdbFilePath "D:\Recovery\Mailbox Database 1882717321.edb" -LogFolderPath "D:\Recovery"

This will cause Exchange to create a new recovery database named RDB1 on server EX1 using the database and logs in D:\Recovery.  Once this command is run, you will see the recovery database in the Exchange Management Console (EMC), but it must be brought into a clean shutdown state before it can be mounted.

To bring the database into a clean shutdown state, use ESEUTIL /R to perform a recovery of the database.  Often, I've seen that Exchange is unable to perform a successful recovery, giving the following error:

Operation terminated with error -1216 (JET_errAttachedDatabaseMismatch, An outstanding database attachment has been detected at the start or end of recovery, but database is missing or does not match attachment info) after 11.625 seconds.

In these cases, I have run an ESEUTIL /P (repair) to force the database into consistency.  Once the database has been successfully recovered or repaired, mount the database in EMC or using the Mount-Database cmdlet.

Now we're ready to recover deleted items from the recovery database.  In order to do this, though, you need Organization Management rights in Exchange 2010.  The following are cmdlet examples for recovering items from the RDB:

This example restores a mailbox for user Keith Johnson, overwriting the existing mailbox:

Restore-Mailbox -ID 'Keith Johnson' -RecoveryDatabase RDB1

This example restores Keith Johnson's mailbox content into an Investigation mailbox:

Restore-Mailbox -ID 'Investigation' -RecoveryDatabase RDB1 -RecoveryMailbox 'Keith Johnson'

This example restores only the mail with the word "contract" in the subject and the word "CompanyABC" in the body of the message from the Inbox or Saved folders.

Restore-Mailbox -ID 'Keith Johnson' -RecoveryDatabase RDB1 -SubjectKeywords 'contract' -ContentKeywords 'companyabc' -IncludeFolders \Inbox,\Saved

There are a lot of different options in the Restore-Mailbox cmdlet and recovery databases that make it a powerful tool for recovery.  Take the time to learn them before you need to use them.

Read more ...

How to Backup Exchange 2010 RTM at Release Timeframe

As with any other major release of Exchange, there will be a gap in third-party vendor support for Exchange 2010 when it is released to general availability next month.

One of those gaps will be supported backup solutions for Exchange 2010.  Thankfully, Microsoft recognized this and added VSS backup support to the built-in Windows Server Backup feature in both Windows Server 2008 and Windows Server 2008 R2.  This capability has been introduced in Exchange 2007 SP2 and Exchange 2010 RTM, allowing you to backup Exchange 2007 SP2 and Exchange 2010 using a native VSS application provider.

Exchange automatically registers its application provider in VSS when Exchange 2010 is installed or when the Exchange 2007 server is upgraded to SP2.  This happens even if the Windows Server Backup feature isn't installed on the server yet.  You simply need to add the Windows Server Backup feature using Server Manager to your Exchange server to enable the Exchange aware VSS backup capability. 

Windows Server Backup (WSB) will allow you to perform Exchange aware backups, similar to NTBackup, with a few notible points:

• Legacy (streaming) backups are not supported.
  
• Since Windows Server Backup performs volume-only Volume Snapshot Service (VSS) backups, there is no specific "Exchange only" backup capability.  When you perform a backup of a volume that contains Exchange data (EDB and log files), WSB automatically performs an Exchange aware backup.  The only visual queue you will see is this, just before the data is backed up:
  

 

• Once WSB notifies Exchange that the VSS Full Backup has completed successfully, Exchange will truncate the log files for all the Exchange 2010 databases or Exchange 2007 SP2 Storage Groups.

Note: The default behavior of WSB is to perform a VSS Copy Backup, which will not truncate the logs. To configure a VSS Full Backup you must configure a Custom backup (not Full Server), add the volumes that contain the Exchange data, click Advanced Settings, and select VSS Full Backup on the VSS Settings tab.

• Backups must be run against the active node on Database Availability Groups (DAGs) or the active node in an Exchange 2007 CCR cluster.  When the backups complete successfully and the logs are truncated on the active node, the same operation will occur on the passive node.
  
• You can backup either to a local hard drive or a network share
  
• There is no remote server backup functionality. You must perform the backup from the Exchange server.
  
• You can schedule the backups using WSB or install the WSB command line extensions to run a backup from the command line.
  
• When restoring, you do not have to restore the whole backed up volume. You can choose to restore only Exchange application data by choosing to recover only the Exchange application, as shown:

And then select Exchange:

• Recovery can be performed to the original location (overwriting the existing data) or to a new folder or location.  If you choose to recover to another location, WSB will copy just the application data, not recover the Exchange application itself.  You can then use this data in an Exchange 2010 Recovery Database (RDB) or an Exchange 2007 Recovery Storage Group (RSG).
  
• You can redirect the restore of an Exchange application to another server.
  
• Microsoft Data Protection Manager (DPM) 2010 is also in beta and is available for download.

In a future article, I will explain the process of using an Exchange 2010 Recovery Database (RDB) to recover data from a backup set.

Read more ...

Windows 7 Interoperability Pack Released

Microsoft announced today the release of the Platform Update for Windows Server 2008 and Windows Vista, as well as Remote Desktop Connection Client 7.0 and Windows Management Framework.  This was previously known as the Windows 7 Interoperability Pack.

Please see the following Microsoft Knowledge Base articles for more information.

• Platform Update for Windows Server 2008 and Windows Vista - http://support.microsoft.com/kb/971644
•  Remote Desktop Connection 7.0 client update - http://support.microsoft.com/kb/969084
• Windows Management Framework Core - http://support.microsoft.com/kb/960930
• Windows Management Framework BITS - http://support.microsoft.com/kb/960568

Read more ...

Windows Management Framework Released

Windows Management Framework, which includes Windows PowerShell 2.0, WinRM 2.0, and BITS 4.0, was officially released to the world this morning. By providing a consistent management interface across the various flavors of Windows, we are making our platform that much more attractive to deploy. IT Professionals can now easily manage their Windows XP, Windows Server 2003, Windows Vista, Windows Server 2008, Windows 7, and Windows Server 2008 R2 machines through PowerShell remoting – that’s a huge win!

You can download the packages here: http://go.microsoft.com/fwlink/?LinkID=151321

Read more ...

Paused Hyper-V VMs Do Not Release RAM

Windows Server 2008 Hyper-V allows the administrator to pause a running Hyper-V virtual machine.  When a VM is paused, the VM system state is written to a file on the host server and the VM no longer will process operations.  This is similar to the sleep feature in other versions of Windows.

When the VM is resumed, Hyper-V will read this saved state information back into its working set and the VM will continue to function as it was when the VM was paused.  This is a very quick operation.

Pausing a VM is handy when you want to quickly and temporarily take a machine offline without shutting it down.  For example, you may want to test cluster failover or you may need to briefly free up main processor resources.

Be aware, however, that pausing a VM does not free up the RAM associated with the VM.  I've seen several customers make this mistake, thinking that they could essentially "over-subscribe" their Hyper-V host server by pausing running VMs to free up resources (RAM) and run other VMs.

When you pause a virtual machine, the RAM allocated to the paused VM is not released back to the host.  Take a look at the sample perfomance monitor screenshot below:

This perfmon example shows available megabytes free on a Windows Server 2008 Hyper-V host server with 8GB RAM.  RAM drops when a 4GB VM is started up, as expected.  The VM is then pause and the available megabytes free remains steady at about 3289MB free.  RAM utilization remains steady when the VM is resumed a short time later.  RAM is only released back to the Hyper-V host when the VM is powered off.

If you want to free up RAM from a running VM, you need to either turn off the VM or use the Hyper-V "Save" action.  Save is similar to the Windows hibernate feature, where both the system state and the RAM working set are written to disk files and then released to the host server.  When the VM is started, it will read these files back into memory and restore the VM to its previous state.

Read more ...

Hyper-V-Worker Event 23012 Explained

If you load a Windows Server 2008 R2 virtual machine on a Windows Server 2008 Hyper-V host server, you will get an error on the host server similar to the following:

Log Name: Microsoft-Windows-Hyper-V-Worker-Admin

Source: Microsoft-Windows-Hyper-V-Worker
Date: 10/23/2009 7:56:48 AM
Event ID: 23012
Task Category: None
Level: Error
Keywords:
User: NETWORK SERVICE
Computer: mailgate.theguillets.com

Description:
Device 'VMBus' in 'EX1 ENT x64' cannot load because it is incompatible with virtualization stack. Server version 13 Client version 65537 (Virtual machine 98EEEED7-A97D-48CF-87F5-E1E8F698D169).

This happens because the Windows Server 2008 R2 Hyper-V Integration Components are not compatible with the Hyper-V v1 release components. 

Incompatible does not mean they won't work - because they do.  It's just that the R2 version includes enhancements and changes that are beyond the capabilities of Hyper-V v1.

If you want to run an R2 build in a VM on Hyper-V v1 and you don't want to see this error, use a Legacy NIC for the R2 VM.

The Integration Components are already present in Windows Server 2008 and Windows Server 2008 R2.  You do not need to install them on these VMs.  You can only upgrade the Integration Components, not downgrade them.

Read more ...

Exchange Server 2010 RTM Upgrade and Installation - Phase 2

These are my notes for phase 2 of my migration from Exchange 2007 SP2 to Exchange 2010 RTM. My notes for phase 1, where I introduced the first Exchange 2010 Hub/CAS/Mailbox server into my existing Exchange 2007 environment, can be read here.

Now in phase 2, I needed to configure the new 2010 server, test mailflow, move the mailboxes, and configure ActiveSync.

I decided to create a third phase, where I will decommission the Exchange 2007 Hub/CAS/Mailbox server, migrate the Windows Server 2008 SP2 Hyper-V host server to Windows Server 2008 R2, and install the Exchange 2010 Edge Transport role on it.

I configured the logging for each server and resubscribed my Edge Transport server. If you don't do this, you'll get the following warning in the Application event log of the 2010 Hub Transport server:

Log Name: Application

Source: MSExchange EdgeSync
Date: 10/22/2009 3:07:25 PM
Event ID: 1032
Task Category: Topology
Level: Warning
Keywords: Classic
User: N/A
Computer: ex1.expta.com

Description:
Microsoft Exchange EdgeSync can't find the replication credential on EX1.expta.com to synchronize with Edge server mailgate.expta.com. This may happen if EX1.expta.com joined the current Active Directory site after subscription for edge.expta.com was established. To have this Hub Transport server participate in EdgeSync, re-subscribe mailgate.expta.com to the current Active Directory site.

There's no need to remove the old subscription. Just create a new subscription file using the New-EdgeSubscription cmdlet on the Edge Transport server and import it using the New Edge Subscription action in EMC on the 2010 Hub Transport server, as usual. It will update the existing Edge subscription for the new 2010 server.

Next, I reconfigured port forwarding for my Client SMTP Send Connector (TCP port 587) to be directed to the new 2010 server. I tested this using my iPhone, which is connected to my home email using IMAP4 and SMTP. In this configuration, the iPhone gets email from the Exchange 2007 server, but sends email through the Exchange 2010 server. Both incoming and outgoing emails tested fine.

Now I needed to move the mailboxes to the new 2010 server. This is accomplished using the Exchange 2010 Management Console to perform Local Move Requests to the database on the 2010 server. Once the move is completed, I cleared the Move Request in the console to complete the move.

Now it was time to move IMAP services to the new 2010 server. As in previous versions of Exchange, the Microsoft Exchange IMAP4 and Microsoft Exchange POP3 services are set to manual and stopped, by default. I changed the Microsoft Exchange IMAP4 service to automatic and started it. Then I reconfigured port forwarding for IMAP4 (TCP port 143) and IMAP4/TLS (TCP port 993) to be directed to the new server. I sync'd the iPhone using secure IMAP and it worked fine.

Note: I use self-signed certificates for Exchange 2007 and 2010. The iPhone will give a warning saying that the certificate may not be trusted. When you continue anyway, the certificate is automatically installed on the iPhone and you won't be prompted again. Cool!

Next, I used the Microsoft Exchange ActiveSync Connectivity Tests in the Microsoft Exchange Remote Connectivity Analyzer to test that ActiveSync is working properly. This tool allows you to remotely test several aspects of you Exchange infrastructure, including Outlook and ActiveSync AutoDiscover records, ActiveSync functionality, Outlook Anywhere, inbound / outbound SMTP email, and more from a Microsoft-hosted website. Very. Very. Cool. The Exchange team just recently updated the ExRCA to work with Exchange 2010.

Here, I ran into an unexpected problem. The ActiveSync tests were failing in ExRCA with the error, "Exchange ActiveSync returned an HTTP 500 response", as shown below.

Unfortunately, the "Tell me more about this issue and how to resolve it" link refers to a less than helpful article for Exchange 2003. I checked the event logs and found the following error in the Application event log:

Log Name: Application

Source: MSExchange ActiveSync
Date: 10/22/2009 9:18:03 PM
Event ID: 1053
Task Category: Configuration
Level: Error
Keywords: Classic
User: N/A
Computer: ex1.expta.com

Description:
Exchange ActiveSync doesn't have sufficient permissions to create the "CN=Keith Johnson,CN=Users,DC=expta,DC=com" container under Active Directory user "Active Directory operation failed on dc1.expta.com. This error is not retriable. Additional information: Access is denied.

Active directory response: 00000005: SecErr: DSID-031521D0, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0
".
Make sure the user has inherited permission granted to domain\Exchange Servers to allow List, Create child, Delete child of object type "msExchangeActiveSyncDevices" and doesn't have any deny permissions that block such operations.

After a bit of research, I discovered that this happens when a user is a member of a Windows built-in group. In my case, the user was a member of Domain Admins. As you probably know, it's best practice to only use admin accounts for administrative functions and to not use them for regular user functions, such as ActiveSync.

To fix the problem, you must remove the user from the built-in group and reconfigure the user's security to apply inheritance (in ADUC, select the Security tab, Advanced, and check Include inheritable permissions from this object's parent). If you don't remove the user from the built-in group, Windows will deselect inheritance.

Once I did all this and retested the ActiveSync functionality using ExRCA, I was ready to configure ActiveSync for my most important user - my wife with her iPhone. It worked like a charm.

There's just a little bit of cleanup to do now. I need to move the Offline Address Book to the new 2010 server and then I can move on to phase 3, where I will decommission the Exchange 2007 server and upgrade the Hyper-V host and Edge Transport server.

Read more ...