Microsoft Exchange 2016 DAG Error Setting Witness Server and WitnessDirectory

We have a number of 2 node DAGs which provide our Microsoft Exchange 2016 service. Being a 2 node DAG a file share witness server is in place. We had a datacentre event that took out one of the nodes and the witness server, rendering the one of the DAGs unhealthy and a second in an odd state.

This article discusses what was observed with the second DAG which was in an odd state. We wanted to ensure that the DAG was healthy, checking the DAG we saw this:

[PS] C:\Windows\system32>Get-DatabaseAvailabilityGroup -Identity DAG03 -status | format-list Name, *Witness*, Servers


Name                      : DAG03
WitnessServer             : exch-wit-b.domain.com
WitnessDirectory          : C:\DAGFileShareWitnesses\DAG03.domain.com
AlternateWitnessServer    : exch-wit-a.domain.com
AlternateWitnessDirectory : C:\DAGFileShareWitnesses\DAG03.domain.com
WitnessShareInUse         : Alternate
DxStoreWitnessServers     :
Servers                   : {EXCH-03A, EXCH-03B}

As you can see the WitnessShareInUse is showing “Alternative” that isn’t the normal state, we wanted to flip the Witness to the primary Server and Directory. So to fix it we ran this following command, which basically instructs the DAG to return to the normal Witness Directory and Server, you can do this running live, assuming both nodes and the witness are available, this command should work to restore the normal state

[PS] C:\Windows\system32>Set-DatabaseAvailabilityGroup -Identity "DAG03" -WitnessServer "exch-wit-b.domain.com" -WitnessDirectory "C:\DAGFileShareWitnesses\DAG03.domain.com"

But this gave:

There was a problem changing the quorum model for database availability group DAG03. Error: An error occurred while attempting a cluster
operation. Error: Cluster API failed: "DeleteClusterResource() failed with 0x139f. Error: The group or resource is not in the correct state
to perform the requested operation"
    + CategoryInfo          : InvalidArgument: (:) [Set-DatabaseAvailabilityGroup], DagTaskProblemChangingQuorumException
    + FullyQualifiedErrorId : [Server=EXCH-03B,RequestId=622ef99d-13fc-4f46-b03d-bdf0c5bf324a,TimeStamp=15/09/2023 10:52:47] [FailureC
   ategory=Cmdlet-DagTaskProblemChangingQuorumException] 357252A0,Microsoft.Exchange.Management.SystemConfigurationTasks.SetDatabaseAvailabi
  lityGroup
    + PSComputerName        : exch-03b.domain.com

Essentially, the EXCH-03B server is having issues, this command should work, its the “not in the correct state” bit which is odd. So we performed a failover of all databases to the EXCH-03A node, then rebooted EXCH-03B, upon it booting back up we tried again to flip the witness and…..

[PS] C:\Windows\system32>Set-DatabaseAvailabilityGroup -Identity "DAG03" -WitnessServer "exch-wit-b.domain.com" -WitnessDirectory "C:\DAGFileShareWitnesses\DAG1603.domain.com"

[PS] C:\Windows\system32>Get-DatabaseAvailabilityGroup -Identity DAG03 -status | format-list Name, *Witness*, Servers


Name                      : DAG03
WitnessServer             : exch-wit-b.domain.com
WitnessDirectory          : C:\DAGFileShareWitnesses\DAG03.domain.com
AlternateWitnessServer    : exch-wit-a.domain.com
AlternateWitnessDirectory : C:\DAGFileShareWitnesses\DAG03.domain.com
WitnessShareInUse         : Primary
DxStoreWitnessServers     :
Servers                   : {EXCH-03A, EXCH-03B}

Success!

What Does the Alternate Witness Server and Alternate Witness Directory Actually Do?

Well, its probably easiest to answer what it doesn’t do first. The Alternate Witness Server and Alternate Witness Directory are not a fail over for your Witness Server.

In a two node DAG (or an even node DAG, e.g 4) you need a “Witness” that can act as a “tie-breaker” in the event of a failure, each node (and witness) has a vote, to be able to claim you are the active Exchange Server in a DAG for a particular database you must have a “majority”, hence the name majority node set.

If say one DAG node (Node A) were to fail, the second DAG node (Node B) and the witness would mean it (Node B) had the majority (i.e. 2 out of 3) and so would become the primary server.

So, the Alternate Witness Server/Directory does not mean you can have two nodes and two witnesses, and if your primary witness were to fail, the second witness (alternate) would kick in automatically and continue to be a Witness. No, this isn’t what it is for. If your primary Witness were to fail, you’re DAG is without a Witness, in this state, if a further failure were to occur, e.g. Node A, then the whole DAG goes offline.

So what does the Alternate Witness Server and Directory do then?

It is there specifically for a scenario where you lose the the ability to have a majority in your DAG, i.e. you lose Node A and the Witness for example. In this situation the “restore-databaseavailabilitygroup” command can be used (see Microsoft link below) to flip to the Alternate Witness and allow the DAG to start up. You might have then when you have a DR setup across sites.

https://learn.microsoft.com/en-us/powershell/module/exchange/restore-databaseavailabilitygroup?view=exchange-ps

You should also consider DAC (Database Availability Coordination) Mode if you are dealing with this type of configuration.

https://learn.microsoft.com/en-us/exchange/high-availability/database-availability-groups/dac-mode?view=exchserver-2019