The question of: I have just ran New-DatabaseAvailabilityGroup and I do not see a File Share Witness (FSW) folder or cluster resources. Where are all my Database Availability Group (DAG) and cluster resources…? What went wrong?
This is a common scenario when an admin is building a DAG, and during the process they check to see that what has changed on the servers. One thing that that causes a false alarm is the lack of a FSW immediately after the DAG has been created, and also a “missing” cluster. Is this something to be concerned about? When are the cluster and FSW resources actually created? Let’s go hit the lab and find out!
In this example, we will be using the default DAG type in Exchange 2013 which is to create the underlying cluster with the Administrative Access Point. Exchange 2013 SP1 and newer introduced the option to create a cluster without the Administrative Access Point when installed on Windows 2012 R2. This has sometimes been called the IP-less DAG, since there are no longer any cluster resources which require the underlying cluster IP.
Lab Starting Configuration
The below is an Exchange 2013 CU12 based lab, where all Exchange servers are running the same build. Domain controllers and the OS on the Exchange servers are all Windows Server 2012 R2. First up, note that there are no DAGs present as we begin this exercise.
Get-DatabaseAvailabilityGroup
On CONSEA-MB1 the Windows Failover Cluster RSAT tool was installed so we can search for any existing clusters. Opening up Failover Cluster Manager, we cannot see any existing clusters.
Create Database Availability Group
Next, let’s create a DAG in PowerShell by running:
New-DatabaseAvailabilityGroup –Name DAG-01 -DatabaseAvailabilityGroupIPAddresses 192.168.2.50
Since there are separate CAS/Mailbox servers in the AD site, Exchange automatically chose to place the FSW onto the CONSEA-CAS2 server. You will note that this was not specified in the above command. The IP address was specified for completeness. The details of the DAG can be seen below.
Get-DatabaseAvailabilityGroup –Identity DAG-01 | Format-List
At this point the DAG exists only in AD as a configuration object. There is no cluster, and now FSW content has been created. For reference the DAG object can be viewed with Get-ADObject
Get-AdObject “CN=DAG-01,CN=Database Availability Groups,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Contoso,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=contoso,DC=com”
Note that there are no servers listed in the started servers list. We can see this via Exchange cmdlets, or by directly querying the underlying AD attributes. The below shows the list of started servers in the DAG.
We saw that the FSW is to be held on the CONSEA-CAS2 server. On that machine the root folder for the witness directories was created, but is empty. This is shown below.
The NTFS ACLs for the directory are below. Note that the Exchange Trusted Subsystem group is present in the local Administrators group as Exchange 2013 was previously installed onto CONSEA-CAS2.
Add First Mailbox Server to the DAG
Since the underlying OS is Windows Server 2012 or newer we must pre-stage the DAG CNO. If you fail to do this, then you will be the lucky winner of this error message: “You Must Provide A Value For This Property”. The pre-stage process and instructions are contained in this post. Once you have done the necessary, return back here to continue.
At this point we have not yet created the underlying cluster, as we have still not added first mailbox server to the DAG. It is the act of adding the first mailbox server to the DAG which will initiate the call to create and build out the cluster. Once AD has replicated with the newly created DAG, we can add the first mailbox server into the DAG. The below process uses PowerShell, though you can use the EAC if you want.
Add-DatabaseAvailabilityGroupServer -Identity DAG-01 -MailboxServer CONSEA-MB1
The comment that it will take <45 seconds is a little optimistic. But once it has done it’s thing all is good.
If there are any errors to check and troubleshoot start with the DAG log file which is located in:
C:\ExchangeSetupLogs\DagTasks
A full path example would be:
C:\ExchangeSetupLogs\DagTasksdagtask_2016-09-15_04-56-51.831_add-databaseavailabiltygroupserver.log
The cluster has now been formed. However note that there is no FSW currently is use as there is only a single server. We can see that in the highlighted section below, where the witness type is set as None.
And the same is noted in Get-DatabaseAvailabilityGroup – the WitnessShareInUse is currently NONE.
The FSW folder is still empty. This is because the FSW is not in use as the cluster is currently in node majority.
Note On IP Address
Remember in the command above to create the DAG, the IP address was specified. If the IP address is not specified, and there is no DHCP available then Failover Cluster Manger will look a little bit angry as the Cluster’s IP address will be in a failed state. Just add the IP address using the Exchange Admin Centre or PowerShell and you will be in business. This assumes that you are not creating the DAG without the Administrative Access Point, or in the vernacular term IP-less DAG.
The below could be used to add one IP4 address to the DAG.
Set-DatabaseAvailabilityGroup –Identity DAG-01 –DatabaseAvailabilityGroupIPAddresses 192.168.2.50
Note that an additional IP will have to be added for the MAPI network if there is also a second secondary site. Simply re-running the above command with the additional IP will overwrite the original IP leaving only the second one present. Both IPs are to be specified. This is shown in the example below”
Set-DatabaseAvailabilityGroup -Identity DAG-01 -DatabaseAvailabilityGroupIpAddresses 192.168.2.50,10.0.1.8
Adding Second Mailbox Server To DAG
The second mailbox server will be added to the DAG. On this server, the Windows Failover Cluster bits were not present so were automatically installed.
After the command completes, the second server is visible using Get-DatabaseAvailabilityGroup. This is highlighted below.
We can also use Cluster.exe /Quorum to have a look at the cluster details.
When the second node is added, Exchange automatically changes the quorum type for us, note that after adding the second node it is now Node and File Share Majority. Since the FSW is in use it's contents can be viewed.
As a side bar you will note that the FSW folder contains a bit of a random number.
C:\DAGFileShareWitnesses\DAG-01.contoso.com49bf4e3d-caa5-4df6-8e2d-38293f9533d7
Well not much in computers is really random. This does look like a GUID. Is it the GUID of the DAG object?
Nope - the folder name is based off the FSW GUID. This can be seen in the cluster log when the resource was created. First we need to generate the cluster log in Windows 2012 and newer. The Get-ClusterLog cmdlet is used for this task.
Get-ClusterLog -Destination C:\Temp\Clusterlog
Looking through the log we see the FSW GUID listed when the it’s cluster resource is created.
The relevant excerpt from the cluster log file is below:
0000312c.00002ca0::2016/09/15-21:58:23.417 INFO [RCM] rcm::RcmApi::CreateResource: (Cluster Group, File Share Witness (\consea-cas2.contoso.comDAG-01.contoso.com), 49bf4e3d-caa5-4df6-8e2d-38293f9533d7, File Share Witness)
Alternatively we can look at the resource’s properties. In this case this we are using PowerShell to retrieve the ID of the FSW:
Get-ClusterResource "File Share Witness (\consea-cas2.contoso.comDAG-01.contoso.com)"
(Get-ClusterResource "File Share Witness (\consea-cas2.contoso.comDAG-01.contoso.com)").ID
Cheers,
Rhoderick
