Agile Sharepoint development by 21apps and Andrew Woodward MVP

Ever wondered what it is like to be on 3rd line support at Microsoft?  The client calls in with a really obscure error that only happens some of the time?  No me either,  but the past few days have probably been pretty close.

The Problem

I am working on a Silverlight/SharePoint solution, that will take the Education world by storm in the not too distant future, which was throwing a very strange error during the creation of a SharePoint web (SPWeb).

   1: MESSAGE

   2: System.NullReferenceException

   3: Object reference not set to an instance of an object.

   4:

   5: STACK TRACE

   6:    at Microsoft.SharePoint.Utilities.SPUtility.StackTraceString(Int32 numLevelsToSkip)

   7:    at Microsoft.SharePoint.SPList.TraceListOperation(String operation, Boolean recordStack)

   8:    at Microsoft.SharePoint.SPList.Update(Boolean bFromMigration)

   9:    at Microsoft.SharePoint.SPList.Update()

  10:    at Microsoft.SharePoint.Workflow.SPWorkflowAssociationCollection.AddCore(SPWorkflowAssociation wa, Guid id, SPList list, Boolean forceUtilityListCreation)

  11:    at Microsoft.SharePoint.Workflow.SPContentTypeWorkflowAssociationCollection.AddCoreCT(SPWorkflowAssociation wa)

  12:    at Microsoft.SharePoint.Workflow.SPContentTypeWorkflowAssociationCollection.PushDownAssociation(SPWorkflowAssociation associationTemplate, Boolean bUpdateIfExisting, MethodBase mbChangeEntry)

  13:    at Microsoft.SharePoint.Workflow.SPContentTypeWorkflowAssociationCollection.CopyWorkflowAssociation(SPWorkflowAssociation associationTemplate)

  14:    at Microsoft.SharePoint.SPContentType.CopyWorkflowAssociationsTo(SPContentType ctDst)

  15:    at Microsoft.SharePoint.SPWeb.SyncNewLists()

  16:    at Microsoft.SharePoint.SPWeb.ApplyWebTemplate(String strWebTemplate)

  17:    at Microsoft.SharePoint.SPWeb.CreateWeb(String strWebUrl, String strTitle, String strDescription, UInt32 nLCID, String strWebTemplate, Boolean bCreateUniqueSubweb, Boolean bConvertIfThere)

  18:    at Microsoft.SharePoint.SPWeb.SPWebCollectionProvider.CreateWeb(String strWebUrl, String strTitle, String strDescription, UInt32 nLCID, String strWebTemplate, Boolean bCreateUniqueSubweb, Boolean bConvertIfThere)

  19:    at Microsoft.SharePoint.SPWebCollection.Add(String strWebUrl, String strTitle, String strDescription, UInt32 nLCID, String strWebTemplate, Boolean useUniquePermissions, Boolean bConvertIfThere)

The problem was happening during the creation of a standard Team Site, although it only happened on some of the SiteCollections and not on others.  Although the error was fired the actual web was actually created, and if you created the site through the UI it did not report and error.

Concern One: The UI is handling and ignoring these errors, if they are not important why would the API through them?

Reading through the errors the problem is rooted in the method Microsoft.SharePoint.Workflow.SPWorkflowAssociationCollection.AddCore which is obfuscated which makes debugging that bit harder.  The method lived deep inside Microsoft.SharePoint.SPWebCollection.Add

Diagnosis

The diagnosis process went through a number of iterations, comparing the sites that worked with those that didn’t. Eventually a small console application was created to look at the types of Work Flow Associations that existed for each of the content types and lists.   The console application accepts one parameter, the web url.

if (args.Length == 0)
{
    Console.WriteLine("Enter URL to web");
}
else
{
    using (SPSite site = new SPSite(args[0]))
    {
        using (SPWeb web = site.OpenWeb())
        {

            Console.WriteLine("***************************");
            Console.WriteLine("Workflow Templates");
            Console.WriteLine("***************************");

            foreach (SPContentType contentType in web.ContentTypes)
            {
                Console.WriteLine(contentType.Name + " - " + contentType.Hidden + " : " + contentType.Scope);
                foreach (SPWorkflowAssociation workflowAssociation in contentType.WorkflowAssociations)
                {
                    Console.WriteLine("WFA:" + workflowAssociation.Name);
                }
            }

            Console.WriteLine("***************************");
            Console.WriteLine("Workflow Templates");
            Console.WriteLine("***************************");

            foreach (SPWorkflowTemplate template in web.WorkflowTemplates)
            {
                Console.WriteLine(template.Name);
            }

            Console.WriteLine("***************************");
            Console.WriteLine("Lists with WF association");
            Console.WriteLine("***************************");

            foreach (SPList list in web.Lists)
            {
                if (list.WorkflowAssociations.Count > 0)
                {
                    Console.WriteLine(list.Title);

                    foreach (SPWorkflowAssociation workflowAssociation in list.WorkflowAssociations)
                    {
                        Console.WriteLine("WFA:" + workflowAssociation.Name);
                    }
                }
            }
        }
    }

    Console.WriteLine("END");
    Console.Read();
}

The output from this application was quite telling but did not provide the exact cause of the problem.  It showed that the Document content type on the sites that worked had no associated workflow.

Whereas the ones on the sites that did not work did have workflows associated with this core content type.

But this just pointed to some Site Collection level features that had been activated on this site that were not on the other.   Deactivating these features did not help.  Note: others have posted warnings about deactivating these types of features, however I was in a proving environment and could easily restore so this was not such a big issue.

The garden path

Continuing the investigation I felt like I was being led up the garden path however it was this path that lead me to the solution.  I wanted to get rid of any workflows and bring the working and non working sites in line.   I needed to remove these workflow associations from the site that was not working to see if the API call would work.

From Site Content Types gallery I chose the Document content type and workflow settings.   I wanted to stop the workflow from being associated with any new document libraries created using this content time, so in effect all libraries.

The error message that is returned from this function is much more informative.

Feature '75a0fea7-1651-4b91-8cdf-3f24e57526ed' for list template '7149' is not installed in this farm.  The operation could not be completed.
Feature '75a0fea7-d31d-491a-9177-f0e461a81e3f' for list template '1010' is not installed in this farm.  The operation could not be completed.
Feature '75a0fea7-d31d-491a-9177-f0e461a81e3f' for list template '1010' is not installed in this farm.  The operation could not be completed.
Feature '75a0fea7-9507-49c7-a473-0ce55c18ef89' for list template '240' is not installed in this farm.  The operation could not be completed.
   at Microsoft.SharePoint.SPContentType.PushDownChanges(CodeToPushDownChangesToDerivedCT derivedCTPushdownCode, CodeToPushDownChangesToListCTs listDerivedCTsPushdownCode, Boolean throwOnSealedOrReadOnly)
   at Microsoft.SharePoint.SPContentType.UpdateWorkflowAssociationsOnChildren(Boolean bGenerateFullChangeList, Boolean bPushdownDerivedCT, Boolean bPushdownListCTs, Boolean bThrowOnSealedOrReadOnly)
   at Microsoft.SharePoint.SPContentType.UpdateWorkflowAssociationsOnChildren(Boolean bGenerateFullChangeList)
   at Microsoft.SharePoint.ApplicationPages.RemWrkflPage.BtnSubmit_Click(Object sender, EventArgs e)
   at System.Web.UI.WebControls.Button.OnClick(EventArgs e)
   at System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument)
   at System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String eventArgument)
   at System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument)
   at System.Web.UI.Page.RaisePostBackEvent(NameValueCollection postData)
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

This function was looking to do very similar things to the failing call above and PushDownChanges to the content type.   However the error message here revealed that I had some missing features.

What do you do when you get a GUID in an error message,  your search for it.  Thank fully Robert Bogue had done one of those obscure but ultimately very helpful posts about the features in his farm.  And even more thankfully Robert had also installed the Fab 40 features into his farm.  As this revealed that the the proving environment these did not exist.  However they had been deployed in Live.

The Root Cause

It is difficult to exactly pinpoint the root cause,  my objective has been to remove the associated workflows to enable the web to be created through the API.   After installation of the Fab 40 WSP based solutions I was able to remove the workflow,  I tried the No New Instances option without success, the setting was applied but still the web would not create.

The Solution

The solution in this case was to remove the workflow,  this will have the consequence of removing all running instances of the workflow.   I hope to spend more time fine tuning the solution so ensure that the minimum of fixing up can be applied.

Lessons Learnt

Management of your proving environment is as critical as live.   Process and rigor should be put in place to ensure that the environment your using meets it’s needs,  in this case it was to mirror live.  Any short cuts that you have to take in order to meet business needs, such as a site collection restore over a full database restore could have implications that have unexpected consequences.

SharePoint is a very complex and often fickle platform,  to support, maintain and nurture your installation will require patience, time and a will to get to know its little foibles. Ultimately it will be rewarding, sometimes it just doesn’t feel like that.