In order for any database availability group (DAG) that has Datacenter Activation Coordination (DAC) mode to be able to mount any databases, the DAG itself much reach automount consensus. In the following blog posts, I discuss and outline situations where automount consensus may not be achieved, resulting in dismounted databases. A DAG in DAC mode which cannot achieve automount consensus will not automatically mount databases, and administrators are blocked from manually mounting them, as well.
Let’s look at an example to illustrate this scenario. In this example, I have a 4-member DAG comprised of servers DAG-1, DAG-2, DAG-3, and DAG-4
DAG-4 experiences a hardware failure, goes down, and is scheduled for replacement. Although the server is offline, it is still a member of the DAG and it’s underlying cluster.
Next, a power failure occurs resulting in DAG-1, DAG-2, and DAG-3 going offline.
After power is restored, DAG-1, DAG-2, and DAG-3 are booted up.
The cluster service is successfully initialized and a cluster is formed. You can verify the nodes are operational by running
Windows 2008:
PS C:\> cluster node
Listing status for all available nodes:
Node Node ID Status
-------------- ------- ---------------------
DAG-4 1 Down
DAG-3 2 Up
DAG-2 3 Up
DAG-1 4 Up
Windows 2008 R2:
PS C:\> Import-Module FailoverClusters
PS C:\> Get-ClusterNode | fl name,state
Name : DAG-1
State : Up
Name : DAG-2
State : Up
Name : DAG-3
State : Up
Name : DAG-4
State : Down
Using the Get-MailboxDatabase cmdlet, we can see that, even though the servers are up and the cluster is running, no databases are mounted:
[PS] C:\>Get-MailboxDatabase -Status | fl name,mounted
Name : Mailbox Database 1252068500
Mounted : False
Name : Mailbox Database 1757981393
Mounted : False
Name : Mailbox Database 1370762657
Mounted : False
Name : Mailbox Database 1511135053
Mounted : False
Attempting to mount a database (with or without the –Force parameter) results in the following error:
[PS] C:\>Mount-Database "Mailbox Database 1252068500"
Couldn't mount the database that you specified. Specified database: Mailbox Database 1252068500; Error code: An Active Manager operation failed. Error An Active Manager operation encountered an error. To perform this operation, the server must be a member of a database availability group, and the database availability group must have quorum. Error: Automount consensus not reached.. [Server: DAG-1.exchange.msft].
+ CategoryInfo : InvalidOperation: (Mailbox Database 1252068500:ADObjectId) [Mount-Database], InvalidOperationException
+ FullyQualifiedErrorId : FD5FDFE8,Microsoft.Exchange.Management.SystemConfigurationTasks.MountDatabase
So what needs to be done to achieve automount consensus and mount the databases? It’s simple: the administrator can run Start-DatabaseAvailabilityGroup. The cmdlet can be run using the –MailboxServer parameter to specify one of the mailbox servers that is functional. After the cmdlet is run, the specified server will have its DACP bit set to 1. The other servers in the DAG will see that the DACP bit on the specified server is set to 1 and this will cause them to set their DACP bits to 1. And once all of the DAG members have DACP set to 1, automount consensus is reached, and the databases can be mounted. Here is an example:
[PS] C:\>Start-DatabaseAvailabilityGroup -Identity DAG -MailboxServer MBX-1
[PS] C:\>Get-MailboxDatabase -Status | fl name,mounted
Name : Mailbox Database 1252068500
Mounted : True
Name : Mailbox Database 1757981393
Mounted : True
Name : Mailbox Database 1370762657
Mounted : True
Name : Mailbox Database 1511135053
Mounted : True
Start-DatabaseAvailabilityGroup can also be used in scenarios where the remaining survivor of a DAG failure is a single member that is using the witness server boot time to achieve automount consensus. In this case the cmdlet forces the member’s DACP bit to 1 and updates the boot times in the registry for future restarts.
As a side note of caution, administrators should be careful when using Start-DatabaseAvailabilityGroup. It is possible that a split brain condition could be created by using this cmdlet to start the wrong DAG members, or it by using it at the wrong stage of a datacenter switchover. To ensure your datacenter switchovers go smoothly, I recommend using the recently released Datacenter Switchover Tool.