I know…I know…you’ll look at the title and say “bah humbug – Microsoft has that documented already…SQUIRREL!“.  To be truthful, yes, Microsoft does have Technet documentation for moving the Central Management Store:

Move the Lync Server 2010 Central Management Server to Lync Server 2013

If you perform a quick web search you’ll find loads of additional results as well, but I can speak from experience that not all steps are accounted for in all of the documentation.  In one way or another, each post has information missing that can leave you scratching your head at why CMS doesn’t seem to replicate anymore and send you into a frantic troubleshooting frenzy.  Please note that the following information does not apply if you are moving CMS between pools that are already paired.  Use this information only if you are going between non-paired pools within the same server version (2013 to 2013, for example) or if you are moving CMS from one server version to another (2010 to 2013, for example).  Hopefully the information presented will help you move CMS in the most pain-free and seamless way possible!

Step 1 – Backup CMS Data

You’ve already got this data backed up right?  The ZIP file stored safely away on a location for a rainy day?  What’s that…you say you back up the databases using T-SQL?….  *shakes head*…

Before you make a single change in your environment, back up the CMS data using the built-in Management Shell cmdlets:

Export-CsConfiguration -Filename "D:\temp\20160112.zip" -Verbose

Take that file and store it outside of your Lync/Skype4B servers.  Make sure it is available in an emergency – it is your get-out-of-jail card in the event things go horribly wrong.

Step 2 – Update Active Directory Groups

Find the RTCUniversalConfigReplicator Universal security group in your Active Directory forest and look at the membership of the group.  Within that group you should see the computer accounts for the front end servers that are currently serving as the CMS replication master:

RTCUniversalServerConfigReplicator-Start

Add the computer accounts of the front end servers that will be the new CMS replication master to the membership of the Active Directory group:

RTCUniversalServerConfigReplicator-Update

Save your changes and wait for Active Directory replication to complete to all sites before moving forward.

Step 3 – Reboot Front Ends

Restart the front end servers you just added into the RTCUniversalConfigReplicator Active Directory group.  Make sure you maintain replica set quorum during reboots by restarting front ends one-at-a-time to ensure that upgrade domains (and replica sets) stay online, thereby not hosing your new, shiny pool.

Why the reboot, you ask?  Two major reasons:

  1. Active Directory access tokens don’t get updated with group membership changes until logoff/logon (reboots for computer accounts). The computer accounts won’t have updated tokens until you restart the servers and allow the access tokens to be updated with the new group membership rights.
  2. The XDS database on the SQL Back End is secured using the RTCUniversalConfigReplicator Active Directory security group.  If the computer accounts do not have access tokens that reflect that group membership, the Master Replicator services can’t access the XDS database to initiate CMS replication.  Having the access tokens updated ahead of time simply speeds up the move process later on and prevents you from having to reboot servers after Step 7 below.
XDSPermissions

Step 4 – Ensure Your User Account Has Required Permissions

The next steps involve actual installation and/or changes to the Lync/Skype4B environment. In order to implement those changes, make sure you’ve got the required permissions for Lync/Skype4BSQL and the pool file share.

Made changes to your user account membership?  Remember to update your access token!

Step 5 – Install CMS Databases on New Pool

Pick a server on the new pool, open the Management Shell with UAC elevation and run one of the following PowerShell cmdlets:

#Named SQL Instance
Install-CsDatabase -CentralManagementDatabase -SQLServerFQDN sqlserver.domain.com -SqlInstanceName sqlinstance -UseDefaultSQLPaths -Report "D:\Temp\Install-CsDatabase-20160112.html" -Verbose

#OR...

#Default SQL Instance Name
Install-CsDatabase -CentralManagementDatabase -SQLServerFQDN sqlserver.domain.com -SqlInstanceName MSSQLSERVER -UseDefaultSQLPaths -Report "D:\Temp\Install-CsDatabase-20160112.html" -Verbose

With the ‘-Verbose’ switch specified, you’ll see loads of information fly by on the screen. If you watch carefully enough then you can follow along with the execution of the T-SQL scripts and see the results as they happen in real-time. When the execution has finished, open the HTML report and make sure all steps in the report show ‘Success’.

Step 6 – Update Topology

Once the databases are successfully installed, topology must be updated to prepare for the new CMS location. From the same server as step 5, open the Management Shell with UAC elevation and run the following PowerShell cmdlet:

Enable-CsTopology -Report "D:\Temp\Enable-CsTopology-20160112.html" -Verbose

With the ‘-Verbose’ switch specified, you’ll see loads of information fly by on the screen. If you watch carefully enough then you can follow along with the execution of the enablement and see the results as they happen in real-time. When the execution has finished, open the HTML report and make sure all steps in the report show ‘Success’.

Note: You may see some ‘Warnings’ for certain environments. I can’t cover every single scenario, but investigate and resolve any ‘Failures’ first. Look at ‘Warnings’ after to determine if there is a true need for further investigation.

Step 7 – Migrate CMS

This step is the big-kahuna and moves the CMS operation to the new pool. Do not attempt this step until you’ve confirmed all previous steps have been completed successfully!

From the same server as step 6, open the Management Shell with UAC elevation and run the following PowerShell cmdlet:

Move-CsManagementServer -Report "D:\Temp\Move-CsManagementServer-20160112.html" -Verbose -Confirm:$FALSE

With the ‘-Verbose’ switch specified, you’ll see loads of information fly by on the screen. If you watch carefully enough then you can follow along with the execution of the move and see the results as they happen in real-time. When the execution has finished, open the HTML report and make sure all steps in the report show ‘Success’.

Note: Investigate any failures of this cmdlet and resolve immediately. The cmdlet touches many objects (AD, Topology, Databases, etc) so proper completion is critical.

Note: This step also performs the update of the RTCUniversalConfigReplicator Active Directory group. We have already completed that step and as a result, no group changes are necessary during the Enable-CsTopology execution.

Note:  From this point on, your topology is in an precarious state.  Master changes cannot be made while CMS is not available.  This means adding pools, creating voice routes, creating policies, etc – all of those features are unavailable because they require the CMS database be available to write the changes to.  Plan this step and the following activities accordingly so that you minimize the overall impact to yourself and your users!

Step 8 – Check for Service Connection Point Update within Active Directory

This step isn’t documented anywhere…until now. When you move the CMS location, the ‘Move’ cmdlet updates a Service Connection Point within Active Directory that tells Lync/Skype4B Servers where to find the CMS master. The SCP object has to be replicated with the updated value (the new pool’s SQL back end) to all AD sites where Lync/Skype4B servers reside before CMS replication will ever be successful.

Note:  This SCP point is also how Topology Builder finds the CMS master and allows you to download a copy of topology, so it is valuable in multiple aspects!

You can view the current configuration of this SCP object against any DC in your environment through the adsiedit.msc MMC and navigating to Configuration>Services>RTC Service>Topology Settings

ADSI-CMSServiceConnectionPoint

Additionally, you can view the current configuration of this SCP object against any DC in your environment through this Management Shell cmdlet:

Get-CsManagementConnection
Get-CsConfigurationStoreLocation
Cmdlet-CMSServiceConnectionPoint
Lync-CMS-CsConfigurationStoreLocation

Active Directory replication is dependent upon intra-site AND inter-site replication schedules.  Intra-site replication is generally very quick (usually less than 60 seconds), but inter-site replication could take anywhere from 15 minutes to…well…much longer. Make sure that the new SCP information has been replicated to all Active Directory sites where your Lync/Skype4B servers reside. Once AD replication of the SCP object is up-to-date, you can move forward.

Step 9 – Start CMS Replication Services

From the server used in Step 7, open the Management Shell with UAC elevation and run the following PowerShell cmdlet:

Start-CsWindowsService -Verbose

When the execution has finished, open the services.msc MMC and ensure that both the Master Replicator Agent and File Transfer Agent services are started.

In the Lync/Skype4B Event Logs, filter for ‘LS Master Replicator Agent Service’ and ‘LS File Transfer Agent Service’, and examine event logs to ensure that the services successfully started and have begun CMS replication processes:

Event 2021
CMS-Event2021Event 1036
CMS-Event1036Event 2033
CMS-Event2033

Note:  As of this point, CMS is operating from the new pool but only via the server used in the steps above.

Step 10 – Monitor CMS Replication

From the server used in step 9, open the Management Shell with UAC elevation and run the following PowerShell cmdlet:

Get-CsManagementStoreReplicationStatus | Select-Object ReplicaFQDN,UpToDate

Ensure that all replicas show ‘TRUE’ for the ‘UpToDate’ status – if so, your CMS replication is working!  As a fail-safe test you can invoke a CMS replication through the following PowerShell cmdlet:

Invoke-CsManagementStoreReplication

Once the replication has been invoked, simply monitor the ‘Get’ cmdlet over the next 3-5 minutes and all replicas should show ‘FALSE’ initially and then flip to ‘TRUE’ as replication completes.

Step 11 – Install CMS Replication Components

With the environment updated to reflect the new pool as CMS master, you must install the CMS components on the remainder of the new pool. To do this, open up the Lync/Skype4B Deployment Wizard  and run ‘Setup or Remove Lync Server components’ on each remaining front end server in the new pool:

CMS-DeploymentWizardBootstrapper

This step is necessary so that bootstrapper installs the software for the Master Replicator Agent and the File Transfer Agent services, which are required for operation of the CMS master.  When the execution of the Step 2 deployment wizard has finished, open the HTML report and make sure all steps in the report show ‘Success’.

Note: Investigate any failures of this cmdlet and resolve immediately. The bootstrapper process touches many objects on the local server so proper completion is critical.

Step 12 – Remove CMS Replication Components

With the environment updated to reflect the new pool as CMS master, you must remove the CMS components from the old pool. To do this, open up the Lync/Skype4B Deployment Wizard and re-run Step 2 on each front end server in the old pool:

CMS-DeploymentWizardBootstrapper

This step is necessary so that bootstrapper removes the software for the Master Replicator Agent and the File Transfer Agent services, which were once used for operation of the CMS master.  When the execution of the Step 2 deployment wizard has finished, open the HTML report and make sure all steps in the report show ‘Success’.

Note: Investigate any failures of this cmdlet and resolve immediately. The bootstrapper process touches many objects on the local server so proper completion is critical.

Step 13 – Remove CMS Databases (Optional)

With the CMS databases moved to a new pool (and new SQL server back end) you can choose to remove the CMS databses (XDS and LIS) from the old pool (and thus old SQL server back end). I list this step as optional as there is no harm in maintaining the databases on the losing pool, especially in scenarios where the losing pool is running the same version of software as the new pool. In that scenario it is entirely possible that you could choose to move CMS back to the original location, so leaving the databases there keeps you from having to reinstall them at a later date.

If you choose to remove the databases, log on to one of the old front end servers, open the Management Shell with UAC elevation and run one of the following PowerShell cmdlets:

#Named SQL Instance
Uninstall-CsDatabase -CentralManagementDatabase -SQLServerFQDN oldsqlserver.domain.com -SqlInstanceName sqlinstance -Report "D:\Temp\Uninstall-CsDatabase-20160112.html" -Verbose

#OR...

#Default SQL Instance Name
Uninstall-CsDatabase -CentralManagementDatabase -SQLServerFQDN oldsqlserver.domain.com -SqlInstanceName MSSQLSERVER -Report "D:\Temp\Uninstall-CsDatabase-20160112.html" -Verbose

With the ‘-Verbose’ switch specified, you’ll see loads of information fly by on the screen. If you watch carefully enough then you can follow along with the execution of the T-SQL scripts and see the results as they happen in real-time. When the execution has finished, open the HTML report and make sure all steps in the report show ‘Success’.

Step 14 – Update Active Directory Groups (Optional)

If you so choose, you may update the RTCUniversalConfigReplicator Universal security group and remove the computer accounts of the front end servers from the old pool. I list this step as optional as there is no harm in maintaining a group configuration with the old front end accounts as members, especially in scenarios where the losing pool is running the same version of software as the new pool. In that scenario it is entirely possible that you could choose to move CMS back to the original location, so leaving the group membership in place keeps you from having to update the membership at a later date.

If you choose to update the group, simply remove the losing front end computer accounts from the group membership and then reboot the losing front end servers to update their access tokens.

Wrapping Up

After all of this, you should have a functional and working CMS replication master.  Cheers!