‘Preliminary Primary FileShareName Parameter is Unusable’ with Lync Server 2013

I’ve been working through a Lync Server 2010 to Lync Server 2013 migration as of recently, and the error below popped up after I upgraded an existing Lync Server 2010 SBA to Lync Server 2013:

Event 32080, LS Storage Service

A queue flush operation has encountered a file error.

Preliminary primary fileShareName parameter: is unusable.  Exception: System.ArgumentNullException: Value cannot be null.
Parameter name: path
     at System.IO.DirectoryInfo..ctor(String path)
     at Microsfot.Rtc.Internal.Storage.Sql.LysDal.ValidateFileShareName(StoreContext ctx, string fileShareName, String timestamp)
Cause:  There may be permission issues to the file share, local file location, temporary directory, or disk is full.

This is the first time I’ve seen this error – ever – so I was a bit perplexed as to what was causing this.  The error involved the Lync Server Storage Service (LYSS) which is not exactly the best published piece of Lync Server (or Skype for Business Server) so finding root cause might be a needle in a haystack.  LYSS, for the curious folks out there, looks like this from a conceptual point of view:

Note:  For more in-depth information about LYSS, see Mattias Kressmark’s blog post on the topic.

Given that the LYSS Database on each registrar is a temporary storage ground for many Lync Server related activities, it didn’t initially make much sense that LYSS was involving a file share. The only file share that could come to my mind was the Front End pool file share which is defined in topology.  When opening up topology and looking at the file share configuration, some alarm bells started to sound:

File server FQDN:  NETBIOS
File share:  SHARE\FOLDER

I’ve changed the true names above to protect the innocent, but the configuration above gives enough of a picture.  To my eyes there were two significant issues:

1. The File server FQDN was not defined as an FQDN.  It was defined as NETBIOS.
2. The File share was not defined as a share.  It was defined as a share\folder.

Nearly all Microsoft documentation is pretty clear on defining the file share as FQDN\Share.  Could it be possible that this is truly the root cause?

A Short Aside

Almost any Lync/Skype4B architect would agree with avoiding #1 above – everything in Topology must be defined as an FQDN.  Or, well, it should be and Topology Builder should help enforce that.  Lync Server 2013 Topology Builder does not validate the File server FQDN format.  Go ahead and try it yourself…you’ll see no validation errors when you configure things with NETBIOS name within Lync Server 2013 Topology Builder.  Try and do that within Skype for Business Topology Builder, however, and you’ll see a validation error saying the file share is not in FQDN format:

Some Lync/Skype4B architects may not completely agree with avoiding #2 above – that you should only define a share name and not share\folder.  Perform a quick internet search and you’ll find plenty of examples of file shares configured this way with no documented issues that result from it.  Additionally, Microsoft makes no specific statement about this issue in a KB article dedicated to documenting errors for unsupported File Share configuration.  Historically speaking, I have never configured a file share as share\folder, so I couldn’t be 100% certain it was unsupported or caused issues.

Aside Over

The SBA seemed to be perfectly functional, with exception to the error at the start of this post, and the front end pool (Lync Server 2013) was also functional (and did not exhibit the error) so I needed to try and remove variables from the equation.  It seemed there to be two potential root causes:

1. File Share Permissions Issues
2. SBA code issues with current file share configuration

Checking #1 proved to be very simple and we were able to confirm permissions were correct:

Group	Permission	Note
RTCHSUniversalServices	Change	Standard Requirement for Lync Server
RTCComponentUniversalServices	Change	Standard Requirement for Lync Server
RTCUniversalServerAdmins	Change	Standard Requirement for Lync Server
RTCUniversalConfigReplication	Change	Standard Requirement for Lync Server

So if it’s truly not the file share permissions it became an issue of trying to determine if the file store configuration simply wouldn’t work with the SBA code.

Digging In…

First step in obtaining more information is through CLS logs.  I already had the AlwaysOn scenario running on the entire environment so gathering the needed data would be easy.  Additionally, the Event 32080 error was only generated once per day and it was almost at the exact same time every day so I could pull CLS information for a very short period of time and more easily find the information. After pulling the data and looking at the Trace logs I was able to find the exact moment when the error was generated:

LyssDal.ValidateFileShareName:lyssdal.cs
Preliminary primary fileShareName parameter: is unusable.  Exception: System.ArgumentNullException: Value cannot be null.

Looking at the second entry above showed that the code began to also access a local NTFS location of ‘C:\ProgramData\Microsoft\Lync Server’.  Examining that folder path showed that, indeed, there was a folder structure with that seemed to be created by the LYSS functionality:

Even better, I had Process Monitor loaded at the time the error was recorded and I could validate through Process Monitor logs that the SBA a) never attempted to contact any UNC-based resources and b) did successfully access the local NTFS folder location above:

Interesting…very interesting…

Based on the information I’d found thus far, it certainly seems like the SBA code doesn’t like something with the file share.  How about the Front End Servers though?  What do they have to say about this, if anything?

After pulling the data and looking at the Trace logs I was able to find the exact moment when the Front End kicked off this process:

LyssDal.ValidateFileShareName:lyssdal.cs
Primary file share location is [\\NETBIOS\SHARE\FOLDER\1-WebServices-29\StorageService]

The Front End has no issues, whatsoever, and validates the share name and begins a process to connect to ‘1-WebServices-29\StorageService\DataExport\Date’.  Examining that folder path showed that, indeed, there was a folder structure with that included a folder for each front end in the pool, created by the LYSS functionality:

Getting Closer…

Given that the Front End servers seemed to be ‘happy’, it certainly seems like the SBA code might have a specific issue with the file share configuration.  The file share is defined in topology and that gets replicated by CMS to each server in topology.  CMS replication was showing UpToDate for all nodes and I even went so far as examining the local XML data within the FE and SBA local databases to make sure they truly were working off the same topology data.  Indeed, each server had the correct data from CMS:

File Server Configuration (both servers)
SBA File Share Configuration (SBA)
Front End File Share Configuration (Front End)

The servers do have correct CMS information and the SBA still doesn’t like what’s configured.  Running out of options in my arsenal, it lead me to a nearly final conclusion:  define a new file store configuration in topology (a supported file store configuration) and migrate content for the Lync Server 2013 infrastructure in topology.  This process has multiple steps and is well documented within these URLs:

1. http://social.technet.microsoft.com/wiki/contents/articles/15374.change-the-file-store-location-for-lync-server-2013-pool.aspx
2. http://ucken.blogspot.com/2014/01/presentation-issues-after-moving-lync.html

The Results Were?

So we went through the process and made the changes.  Updated Topology to use a correctly formatted file store and updated all related configuration items.  Restarted services.  Following all the changes, I was able to validate…failure.  The change didn’t seem to have any impact and the errors persist today.  A big thanks goes out to Amanda Debler for confirming that she too sees these errors in her SBA event logs even though the file store in that environment is specified in the recommended manner.

Bottom line:  I have no idea why the error appears nor any idea of negative ramifications the error indicates.  Sadly I don’t have a fix for this one…yet.  Even so, this customer would have had to change their file store configuration anyway since the Skype4B Topology Builder wouldn’t allow them to use the current one due to the NETBIOS name configuration.  Not a complete loss, but not the home-run I was hoping for.  🙁

Note:  The SBA was running January 2016 Cumulative Update for Lync Server 2013.

Note:  If you want to manually invoke this LyssDal.Cs process, run the Invoke-CsStorageServiceFlush cmdlet specifying a flushtype of “FullFlush”.  This cmdlet will generate the error on command.