EzCloudInfo

During a DR activation the Activation went fine. But when trying to restore the main site after the DR tests are complete were getting the below error

Below was the Current state in the DR site before the restoration to main site :

Version of Exchange – Exchange 2016 CU3 with no coexistence

1) Main site was in stopped state for DAG and All main site exchange
servers were in Stopped mailbox servers list.
2) DR site was activated for DAG and only DR site exchange servers were in started mailbox servers and operational servers list.
3) All the DR copies were mounted , and users were connected.

After the DR tests were completed and trying to start the main site with below command was getting the below error :

Start-DatabaseAvailabilityGroup -ActiveDirectorySite  “MainSite”

A server-side database availability group administrative operation failed. Error The operation failed. CreateCluster errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Cluster API ‘”AddClusterNode() (MaxPercentage=12) failed with 0x35. Error: The network path was not found”‘

Had a look at the DAG tasks logs and was getting the same above message :

Error: A server-side database availability group administrative operation failed. Error The operation failed. Create Cluster errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Cluster API failed: “AddClusterNode() (MaxPercentage=12) failed with 0x35. Error: The network path was not found”.

Additionally was getting this message in the DAG Task logs

WriteError! Exception = Microsoft.Exchange.Management.Tasks.FailedToStartNodeException: Start-DatabaseAvailabilityGroup failed to start server

Solution :

Followed the below blog steps and it worked :

https://amagsmb.wordpress.com/2015/09/16/problem-adding-a-second-server-to-dag-error-cluster-api-addclusternode-maxpercentage12/

The Remote Registry Service should have the Startup type set to Automatic and be started.
An SP, Windows update or RU installation will put the service in a disabled state and it might be in that state after the update. In my case the main site servers OS were patched last week and post the patches these services might have gone to disabled state. While stopping  and evicting these nodes on activating the DR site there were no issues this was strange and went smooth without any issues.

The real issue happened  only when we tried to activate the main site on re adding these servers back to the DAG group.

Reason:

Some Exchange EMS/Power Shell functions, such as managing diagnostics logging requires the remote registry service to be enabled. So the Exchange required this service on the  remote servers to add them on the node. If this service is not started then the servers will not join in the DAG.

Thanks & Regards
Sathish Veerapandian

During a fail over DR cases when the Main site is completely not available we need to carry over few steps to activate Exchange Services according to the type of DR setup we have.

Sequential steps needs to be carried over in terms of  restoring the DAG,activating the DB’s on the DR site pointing the exchange DNS records to the DR site ip’s.

Failover scenarios varies according to the namespaces, no of sites in Exchange :

UnBound Name Space- Single name space for all Exchange URL’s for both the main and DR sites which is best recommended.
Bound Name Space – Very complicated and not recommended since we need to use seperate URL’s for Main and DR site.

If we have a three site setup with FSW in third site or if the FSW is placed in the Azure directory in the 3rd site then no manual activation of the database copies on the DR site is required. Only exchange DNS job on the DR site is required.

For detailed information on DAG DR setup i have written a previous blog which can be referred:

https://exchangequery.com/2016/05/04/dag-in-exchange-2016-and-windows-server-2012-r2/

From Exchange 2013 the Dynamic Quorum in the failover cluster adjusts automatically and recalculates the active nodes if its on a sequential shutdown for a two site setup.

During a DR activation in the DR site when the main site is completely not available after rebuilding the DAG cluster on the DR site we might come across the below error for some databases

In my test case it was the below:

Stop-DatabaseAvailablityGroup – for the Main site completed successfully with no errors
Restore-DatabaseAvailabilityGroup – completed successfully except some warnings for one mailbox node on the DR site.

On the server with warning noticed that all the DB’s were in failed state.Tried to mount them and got the below error

An Active Manager operation failed. Error The database action failed. Error: The database was not mounted because its experienced data loss as a result of a switchover or failover, and the attempt to copy the last logs from the sourcserver failed. Please check the event log for more detailed information. Specific error message: Attempt to copy remaing log files failed for database DBNAME. Error: Microsoft.Exchange.Cluster.Replay.AcllUnboundedDatalossDetectedEeption:

By looking into the above message its very interesting to see that the DR site DB’s are trying to reach the Main site copies to the get the information though the DAG cluster is activated on the DR site and the PAM is on the DR.

The below command can be used just in case if the DR copies are not mounted after activating the DR site DAG.

Move-ActiveMailboxDatabase “DBNAME” -ActivateOnServer DRMailboxServer -SkipHealthChecks -SkipActiveCopyChecks -SkipClientExperienceChecks -SkipLagChecks -MountDialOverride:besteffort

So we need to be very clear that this error will not occur normally until and unless there is some data loss for any DB’s during the DAG DR activation.

Usually when we do a Restore-DatabaseAvailabilitygroup on the DR site all the DB’s should be mounted on the DR site.

The above command can be run only if the database copies are in a failed state after DR site activation and if they are not getting  mounted.

Thanks & Regards
Sathish Veerapandian
MVP – Office Servers & Services