I know…I know…you’ll look at the title and say “bah humbug – Microsoft has that documented already…SQUIRREL!“. To be truthful, yes, Microsoft does have Technet documentation for moving the Central Management Store:
Move the Lync Server 2010 Central Management Server to Lync Server 2013
If you perform a quick web search you’ll find loads of additional results as well, but I can speak from experience that not all steps are accounted for in all of the documentation. In one way or another, each post has information missing that can leave you scratching your head at why CMS doesn’t seem to replicate anymore and send you into a frantic troubleshooting frenzy. Please note that the following information does not apply if you are moving CMS between pools that are already paired. Use this information only if you are going between non-paired pools within the same server version (2013 to 2013, for example) or if you are moving CMS from one server version to another (2010 to 2013, for example). Hopefully the information presented will help you move CMS in the most pain-free and seamless way possible!
Step 1 – Backup CMS Data
You’ve already got this data backed up right? The ZIP file stored safely away on a location for a rainy day? What’s that…you say you back up the databases using T-SQL?…. *shakes head*…
Before you make a single change in your environment, back up the CMS data using the built-in Management Shell cmdlets:
Export-CsConfiguration -Filename "D:\temp\20160112.zip" -VerboseTake that file and store it outside of your Lync/Skype4B servers. Make sure it is available in an emergency – it is your get-out-of-jail card in the event things go horribly wrong.
Step 2 – Update Active Directory Groups
Find the RTCUniversalConfigReplicator Universal security group in your Active Directory forest and look at the membership of the group. Within that group you should see the computer accounts for the front end servers that are currently serving as the CMS replication master:
Add the computer accounts of the front end servers that will be the new CMS replication master to the membership of the Active Directory group:
Save your changes and wait for Active Directory replication to complete to all sites before moving forward.
Step 3 – Reboot Front Ends
Restart the front end servers you just added into the RTCUniversalConfigReplicator Active Directory group. Make sure you maintain replica set quorum during reboots by restarting front ends one-at-a-time to ensure that upgrade domains (and replica sets) stay online, thereby not hosing your new, shiny pool.
Why the reboot, you ask? Two major reasons:
- Active Directory access tokens don’t get updated with group membership changes until logoff/logon (reboots for computer accounts). The computer accounts won’t have updated tokens until you restart the servers and allow the access tokens to be updated with the new group membership rights.
- The XDS database on the SQL Back End is secured using the RTCUniversalConfigReplicator Active Directory security group. If the computer accounts do not have access tokens that reflect that group membership, the Master Replicator services can’t access the XDS database to initiate CMS replication. Having the access tokens updated ahead of time simply speeds up the move process later on and prevents you from having to reboot servers after Step 7 below.
Step 4 – Ensure Your User Account Has Required Permissions
The next steps involve actual installation and/or changes to the Lync/Skype4B environment. In order to implement those changes, make sure you’ve got the required permissions for Lync/Skype4B, SQL and the pool file share.
Made changes to your user account membership? Remember to update your access token!
Step 5 – Install CMS Databases on New Pool
Pick a server on the new pool, open the Management Shell with UAC elevation and run one of the following PowerShell cmdlets:
#Named SQL Instance Install-CsDatabase -CentralManagementDatabase -SQLServerFQDN sqlserver.domain.com -SqlInstanceName sqlinstance -UseDefaultSQLPaths -Report "D:\Temp\Install-CsDatabase-20160112.html" -Verbose #OR... #Default SQL Instance Name Install-CsDatabase -CentralManagementDatabase -SQLServerFQDN sqlserver.domain.com -SqlInstanceName MSSQLSERVER -UseDefaultSQLPaths -Report "D:\Temp\Install-CsDatabase-20160112.html" -VerboseWith the ‘-Verbose’ switch specified, you’ll see loads of information fly by on the screen. If you watch carefully enough then you can follow along with the execution of the T-SQL scripts and see the results as they happen in real-time. When the execution has finished, open the HTML report and make sure all steps in the report show ‘Success’.
Step 6 – Update Topology
Once the databases are successfully installed, topology must be updated to prepare for the new CMS location. From the same server as step 5, open the Management Shell with UAC elevation and run the following PowerShell cmdlet:
Enable-CsTopology -Report "D:\Temp\Enable-CsTopology-20160112.html" -VerboseWith the ‘-Verbose’ switch specified, you’ll see loads of information fly by on the screen. If you watch carefully enough then you can follow along with the execution of the enablement and see the results as they happen in real-time. When the execution has finished, open the HTML report and make sure all steps in the report show ‘Success’.
Note: You may see some ‘Warnings’ for certain environments. I can’t cover every single scenario, but investigate and resolve any ‘Failures’ first. Look at ‘Warnings’ after to determine if there is a true need for further investigation.
Step 7 – Migrate CMS
This step is the big-kahuna and moves the CMS operation to the new pool. Do not attempt this step until you’ve confirmed all previous steps have been completed successfully!
From the same server as step 6, open the Management Shell with UAC elevation and run the following PowerShell cmdlet:
Move-CsManagementServer -Report "D:\Temp\Move-CsManagementServer-20160112.html" -Verbose -Confirm:$FALSEWith the ‘-Verbose’ switch specified, you’ll see loads of information fly by on the screen. If you watch carefully enough then you can follow along with the execution of the move and see the results as they happen in real-time. When the execution has finished, open the HTML report and make sure all steps in the report show ‘Success’.
Note: Investigate any failures of this cmdlet and resolve immediately. The cmdlet touches many objects (AD, Topology, Databases, etc) so proper completion is critical.
Note: This step also performs the update of the RTCUniversalConfigReplicator Active Directory group. We have already completed that step and as a result, no group changes are necessary during the Enable-CsTopology execution.
Note: From this point on, your topology is in an precarious state. Master changes cannot be made while CMS is not available. This means adding pools, creating voice routes, creating policies, etc – all of those features are unavailable because they require the CMS database be available to write the changes to. Plan this step and the following activities accordingly so that you minimize the overall impact to yourself and your users!
Step 8 – Check for Service Connection Point Update within Active Directory
This step isn’t documented anywhere…until now. When you move the CMS location, the ‘Move’ cmdlet updates a Service Connection Point within Active Directory that tells Lync/Skype4B Servers where to find the CMS master. The SCP object has to be replicated with the updated value (the new pool’s SQL back end) to all AD sites where Lync/Skype4B servers reside before CMS replication will ever be successful.
Note: This SCP point is also how Topology Builder finds the CMS master and allows you to download a copy of topology, so it is valuable in multiple aspects!
You can view the current configuration of this SCP object against any DC in your environment through the adsiedit.msc MMC and navigating to Configuration>Services>RTC Service>Topology Settings
Additionally, you can view the current configuration of this SCP object against any DC in your environment through this Management Shell cmdlet:
Get-CsManagementConnectionGet-CsConfigurationStoreLocationActive Directory replication is dependent upon intra-site AND inter-site replication schedules. Intra-site replication is generally very quick (usually less than 60 seconds), but inter-site replication could take anywhere from 15 minutes to…well…much longer. Make sure that the new SCP information has been replicated to all Active Directory sites where your Lync/Skype4B servers reside. Once AD replication of the SCP object is up-to-date, you can move forward.
Step 9 – Install CMS Replication Components
With the environment updated to reflect the new pool as CMS master, you must install the CMS components on the server in the new pool. To do this, open up the Lync/Skype4B Deployment Wizard and run ‘Setup or Remove Lync Server components’ on the Front End in the new pool you have previously used:
This step is necessary so that bootstrapper installs the software for the Master Replicator Agent and the File Transfer Agent services, which are required for operation of the CMS master. When the execution of the Step 2 deployment wizard has finished, open the HTML report and make sure all steps in the report show ‘Success’.
Note: Investigate any failures of this cmdlet and resolve immediately. The bootstrapper process touches many objects on the local server so proper completion is critical.
Step 10 – Start CMS Replication Services
From the server used in Step 7, open the Management Shell with UAC elevation and run the following PowerShell cmdlet:
Start-CsWindowsService -VerboseWhen the execution has finished, open the services.msc MMC and ensure that both the Master Replicator Agent and File Transfer Agent services are started.
In the Lync/Skype4B Event Logs, filter for ‘LS Master Replicator Agent Service’ and ‘LS File Transfer Agent Service’, and examine event logs to ensure that the services successfully started and have begun CMS replication processes:
Event 2021Event 1036
Event 2033
Note: As of this point, CMS is operating from the new pool but only via the server used in the steps above.
Step 11 – Monitor CMS Replication
From the server used in step 9, open the Management Shell with UAC elevation and run the following PowerShell cmdlet:
Get-CsManagementStoreReplicationStatus | Select-Object ReplicaFQDN,UpToDateEnsure that all replicas show ‘TRUE’ for the ‘UpToDate’ status – if so, your CMS replication is working! As a fail-safe test you can invoke a CMS replication through the following PowerShell cmdlet:
Invoke-CsManagementStoreReplicationOnce the replication has been invoked, simply monitor the ‘Get’ cmdlet over the next 3-5 minutes and all replicas should show ‘FALSE’ initially and then flip to ‘TRUE’ as replication completes.
Step 12 – Install CMS Replication Components
With the environment updated to reflect the new pool as CMS master, you must install the CMS components on the remainder of the new pool. To do this, open up the Lync/Skype4B Deployment Wizard and run ‘Setup or Remove Lync Server components’ on each remaining front end server in the new pool:
This step is necessary so that bootstrapper installs the software for the Master Replicator Agent and the File Transfer Agent services, which are required for operation of the CMS master. When the execution of the Step 2 deployment wizard has finished, open the HTML report and make sure all steps in the report show ‘Success’.
Note: Investigate any failures of this cmdlet and resolve immediately. The bootstrapper process touches many objects on the local server so proper completion is critical.
Step 13 – Remove CMS Replication Components
With the environment updated to reflect the new pool as CMS master, you must remove the CMS components from the old pool. To do this, open up the Lync/Skype4B Deployment Wizard and re-run Step 2 on each front end server in the old pool:
This step is necessary so that bootstrapper removes the software for the Master Replicator Agent and the File Transfer Agent services, which were once used for operation of the CMS master. When the execution of the Step 2 deployment wizard has finished, open the HTML report and make sure all steps in the report show ‘Success’.
Note: Investigate any failures of this cmdlet and resolve immediately. The bootstrapper process touches many objects on the local server so proper completion is critical.
Step 14 – Remove CMS Databases (Optional)
With the CMS databases moved to a new pool (and new SQL server back end) you can choose to remove the CMS databses (XDS and LIS) from the old pool (and thus old SQL server back end). I list this step as optional as there is no harm in maintaining the databases on the losing pool, especially in scenarios where the losing pool is running the same version of software as the new pool. In that scenario it is entirely possible that you could choose to move CMS back to the original location, so leaving the databases there keeps you from having to reinstall them at a later date.
If you choose to remove the databases, log on to one of the old front end servers, open the Management Shell with UAC elevation and run one of the following PowerShell cmdlets:
#Named SQL Instance Uninstall-CsDatabase -CentralManagementDatabase -SQLServerFQDN oldsqlserver.domain.com -SqlInstanceName sqlinstance -Report "D:\Temp\Uninstall-CsDatabase-20160112.html" -Verbose #OR... #Default SQL Instance Name Uninstall-CsDatabase -CentralManagementDatabase -SQLServerFQDN oldsqlserver.domain.com -SqlInstanceName MSSQLSERVER -Report "D:\Temp\Uninstall-CsDatabase-20160112.html" -VerboseWith the ‘-Verbose’ switch specified, you’ll see loads of information fly by on the screen. If you watch carefully enough then you can follow along with the execution of the T-SQL scripts and see the results as they happen in real-time. When the execution has finished, open the HTML report and make sure all steps in the report show ‘Success’.
Step 15 – Update Active Directory Groups (Optional)
If you so choose, you may update the RTCUniversalConfigReplicator Universal security group and remove the computer accounts of the front end servers from the old pool. I list this step as optional as there is no harm in maintaining a group configuration with the old front end accounts as members, especially in scenarios where the losing pool is running the same version of software as the new pool. In that scenario it is entirely possible that you could choose to move CMS back to the original location, so leaving the group membership in place keeps you from having to update the membership at a later date.
If you choose to update the group, simply remove the losing front end computer accounts from the group membership and then reboot the losing front end servers to update their access tokens.
Wrapping Up
After all of this, you should have a functional and working CMS replication master. Cheers!
