Agile Sharepoint development by 21apps and Andrew Woodward MVP

One of my favourite tools Typemock has just been released in a fantastic ASP.Net bundle and have once again offered this up to the blogging community to get for free.

Unit Testing ASP.NET? ASP.NET unit testing has never been this easy.
Typemock is launching a new product for ASP.NET developers – the ASP.NET Bundle – and for the launch will be giving out FREE licenses to bloggers and their readers.
The ASP.NET Bundle is the ultimate ASP.NET unit testing solution, and offers both Typemock Isolator, a unit test tool and Ivonna, the Isolator add-on for ASP.NET unit testing, for a bargain price.
Typemock Isolator is a leading .NET unit testing tool (C# and VB.NET) for many ‘hard to test’ technologies such as SharePoint, ASP.NET, MVC, WCF, WPF, Silverlight and more. Note that for unit testing Silverlight there is an open source Isolator add-on called SilverUnit.
The first 60 bloggers who will blog this text in their blog and tell us about it, will get a Free Isolator ASP.NET Bundle license (Typemock Isolator + Ivonna). If you post this in an ASP.NET dedicated blog, you’ll get a license automatically (even if more than 60 submit) during the first week of this announcement.
Also 8 bloggers will get an additional 2 licenses (each) to give away to their readers / friends.
Go ahead, click the following link for more information on how to get your free license.

I’ve not had the chance to try out the Ivonna or SilverUnit add-ons to the framework yet, but will be looking at these in the near future and they can help you test more of your SharePoint solutions.

If your the only member of the team is it really worth going through the same agile development process that you would in a team of eight?

Isn’t it quicker to just crack open Visual Studio and start coding?   After all your the customer so you know what you want right?   You not going to have to deal with any of the social complexities that conspire against you. You’re going have the hottest of communication with the dev team,  the dev and test roles will have an interment knowledge and the scrum master won’t really have to do anything as there is not going to be any blockers on this project.

Right?

So I started down this path and before I knew it I had some weird project created with bits of code and SharePoint artefacts but not really seeming to make any progress.   I was easily distracted by other things and didn’t really have a feel for how long it would take and when I could say I was done.

STOP!

What was I doing!  I spend my time trying to help people develop better, to take small steps and sometimes big steps to help produce better software..   why was I not practicing what I preached?

I stopped!

Deep breath, deleted all the rubbish I can previously created and got back to basics.

The Team

I have a team,  the fact that some (in this case all) of the roles were done by the same person was ok; that happens often in agile teams.

Design

Yes, you heard me right here.  Eric Shupps would be proud of me .   I’m not going to jump into Visual Studio and crank out code starting [TestFixture] I need to do some design work first, even though this application is going to be small I still need to put in the time on the upfront design.  I need to think through the solution at a high level.

Here I used the opportunity to try out the great Balsamiq application,  which I stumped up the $79 license fee for and download the desktop version.

I’ve deliberately blurred the image but you get the idea of what the mock ups look like, and these take only a matter of minutes to produce.  Definately something you should consider if your currently trying to do this in Visio or heaven forbid cranking out HTML mockups.

The few UI based mockups actually helped me to understand how the application would work for the end user, which in turn provided a clear vision for the way the solution would be developed.

User Stories

I didn’t need to do any remote collaboration on the User Stories so I was able to get back to basics and use one of the white boards in my office as the task board.   I wrote the user stories out on post-it notes and added them to the board along with rough estimates for each one.

Here you can see I’m using a very common step of status on my task board

Backlog - This is where I add all stories that came out of the design,  they are all user focused with nothing specific about the implementation.  No stories like ‘Use jQuery to make it nice’  these are user stories.  If I think of something while i’m coding or testing, or even having a beer I can just write it to a post-it and stick it on the backlog.

Sprint – I like Scrum so adopt a Scrum Esq. style to my process.  The sprint will be where I put the stories I plan to do next.  The amount is based on the estimate and also the sprint length.   In this project I’m doing stupily short sprints of 2 days each.

In Progress – When I pick off a story to work on it goes in here,  if you’ve ever looked at the Kanban approaches your looking to keep this ‘In Progress’ to a defined level.  I like to try and move things through to ‘Done’ before picking off new stories.

Verify – When I think it’s been coded properly and dev tested I move the story here.   Verify is where I would validate the code on another environment – i.e. a non-dev box,  one with multiple web front ends.  At this point I also look at introducing some form of automation for both the build/deployment and also the acceptance testing.

Done – This is when it is really really done,  there is nothing left to do, no documentation, no tests or tweaks,  the code is production ready.

Sprint 1 – Day 1

Having selected the user stories for the sprint it was time to setup my dev project and get some of the foundations in place.   If you follow me on twitter you will know that I previously spent time producing a decent SharePoint development Virtual Machine so it was very quick for me to roll out the few things I needed to get going.   A SharePoint team site in which I could spike and test my code,  a VS2008 solution and class library project in which to start cranking out some code.

At the end of Day 1 my board looked like this:

One of the stories I felt was ready for verification and another I was making good progress on.   At this point I had nothing ‘Done’ my burndown in the morning would look a bit sad,  but that’s OK that is often the case at the start of a project or sprint.  I also don’t at this point have a verification setup so this is going to slow me down a bit on Day 2.

Day 1 has been good,  I have the basic project structure defined.   I was able to build a WSP with the SharePoint features I needed to complete the first story and was able to deploy and test this on my dev machine.

People that have read my blog or seen me talk at conferences and user groups know that I am very keen on doing things right; I like to see Unit Tests and better still I want to see people at least trying Test Driven Development.   However at the end of Day 1 I was yet to write a single line of C# code, neither in test form or production code.   So did I really do development on day one?

Well I think I did, only the language used was XML and a little bit of CAML and for these items I do not know of any way, that makes sense, to unit test.  I often think that the XML we define as part of SharePoint projects are more inline with doing configuration,  we are just setting up SharePoint using a specific configuration and the testing for this should naturally falls into the integration space and will be picked up during verification.   That is not to say as a developer I do not validate it,  I need to ensure that it works as expected in my environment before I can move it over to the Verify column and it’ is the job of the tester to make sure this happens.  For this project the tester (me) was adamant that it all worked as they didn’t want to have to waste time on silly errors.

Interesting Test

One thing that I will be having to do with this project is fit it around real work, the sort that helps to keep food on the table sort of work.   This means that the sprints although 2 days in length the elapsed time may be significantly longer.  I know that this will impact on the velocity as I will lose some of the momentum during these breaks, however the benefit I have from adopting a more defined process is that it will be significantly easier for me to pick up the project.   Imaging if I had to go back to the mess of code I had originally created after a weeks break!

I’m really interested to see if my agile development with a team of one really does make a significant impact on the way I develop the code, on the time it takes to complete it and on the quality of the solution created at the end.

Find out what happens on Day 2.

As development practices around the SharePoint platform mature the demands for better quality and better performing solutions are being made.  Gone are the days when Enterprises will accept that the Roll-Up Web Part they purchased will not scale beyond a few site collections.  A knowledge of the platform and defensive coding alone will not cut the mustard, the need for a holistic approach to your development practices will be the only way to achieve success.  In this blog post I am focusing on one specific technique, code profiling, and using the excellent Red Gate ANTS Profiler.

Disclaimer:  Red Gate ANTS Profiler is a paid for application.  There are other products on the market that do code profiling including Visual Studio.  The reason for using ANTS in this post is due to the ease with which you can get up an running and the quality of the user experience.  I recommend you set aside a day to look at the profiling in Visual Studio and then a few hours to get that much further with ANTS.

Software Performance Analysis – Wikipedia

In software engineering, performance analysis, more commonly today known as profiling, is the investigation of a program’s behavior using information gathered as the program executes (i.e. it is a form of dynamic program analysis, as opposed to static code analysis). The usual goal of performance analysis is to determine which sections of a program to optimize – usually either to increase its speed or decrease its memory requirement (or sometimes both).

As the quote says this technique is specifically aimed at the optimization of code; the tools enable the developer to quickly target the areas of code that need optimization and lead to an improved return for the effort. (seems strange saying ROI when talking about dev techniques, but this is really what it is).

Getting started

Red Gate provide a 14 day trial, so you can try before you buy.  The latest release, version 4.3 as at the time of writing, has made some great improvements over the 4.1 version – most notably the removal of the need to stop all the IIS web sites listening on port 80 .

As you would expect this post will focus on profiling SharePoint code, I’m going to use the Magic 8 ball application developed for my TDD session at the Best Practices SharePoint Conference in San Diego.

More details on Test Driven SharePoint development can be in the Agile section of this site.

Environment

The environment that I will do the testing is a typical SharePoint developers Virtual Machine.

1. Windows 2008 Server (32bit)
2. Visual Studio 2008
3. Microsoft Office SharePoint Server 2007
4. SQL Server 2005

As this is running on Windows 2008 I will be testing against IIS7.   The requirements are identical for Windows 2003 and IIS 6.0

Installation

Download ANTS Profiler and install, not too many options to worry about here.

Luckily for my first run through the team have provided a technical paper on installing, using and getting ANTS Profiler to work with SharePoint, How to profile a SharePoint 2007 Site Collection.  There were a few things in here that I prefer not to do,  I will document what I do below and will pass this back – I’m not criticizing, there’s no way I would even attempt to write a Profiler so mucho respect from me to the Red Gate team

Configuration

We are looking for the ability to profile any site collection within a Web Application.  This involves getting the account used for the associated application pool, giving it some additional rights and then configuring the ANTS services to use this account.

Web Application

Depending on your development machine configuration you will have a number of web applications configured in SharePoint, the configuration below should be reasonably familiar to most of you.

The key here is to identify the web application (IIS Website) as we need to get the account name from the associated Application Pool.

Load IIS Manager, select the web application and chose Basic Settings from the Actions menu on the right.  This will show the associated application pool as below.

Select the Application Pools from the connections menu, choose the application pool identified in the previous step and click advanced settings.  This will give you a properties screen showing the account we need to work with.

In this case 21apps\svc-intranet-ap – if you have a local installation this may be set to Local System in which case the steps below may not be required.   I recommend that dev’s don’t have a default install as you really need to understand how SharePoint is setup to undertake any large scale developments.

Web.Config

In IIS 6 days you would need to edit the web.config file to set the application into debug mode.   In IIS 7 the common ASP.NET configuration options have been promoted to the IIS console.  With the web application still selected click on the .NET Compilation option and change the Debug behaviour to True and click Apply.

Like all good SharePoint developers you will be working with WSS_Minimal trust settings or a custom policy file.  To do profiling however you will need to set the Trust level to Full.  Again IIS 7 provides a UI or you can edit it manually.

It is possible to run the code without setting a Full trust level, but you need to remove the .pdb files and will lose the ability to profile the lines of code.  Which is sort of the point of profiling!  If you don’t set Full trust you will get an error like the one below when loading a web part

Web Part Error: An error has occurred.

Show Error Details
Hide Error Details 

[TargetInvocationException: Exception has been thrown by the target of an invocation.]
  at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandle& ctor, Boolean& bNeedSecurityCheck)
  at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean fillCache)
  at System.RuntimeType.CreateInstanceImpl(Boolean publicOnly, Boolean skipVisibilityChecks, Boolean fillCache)
  at System.RuntimeType.CreateInstanceImpl(Boolean publicOnly)
  at System.Activator.CreateInstance(Type type, Boolean nonPublic)
  at System.Activator.CreateInstance(Type type)
  at Microsoft.SharePoint.WebPartPages.SPWebPartSerializer.get_DefaultControl()
  at Microsoft.SharePoint.WebPartPages.BinaryWebPartSerializer.SerializedWebPart..ctor(SPWebPartManager webPartManager, XmlNamespaceManager xmlnsManager, Byte[] sharedData, Byte[] userData, String[] links, Type type, SPWeb spWeb)
  at Microsoft.SharePoint.WebPartPages.BinaryWebPartSerializer.Deserialize(SPWebPartManager webPartManager, XmlNamespaceManager xmlnsManager, Byte[] userData, Byte[] sharedData, String[] links, Type type, SPWeb spWeb)
  at Microsoft.SharePoint.WebPartPages.SPWebPartManager.CreateWebPartsFromRowSetData(Boolean onlyInitializeClosedWebParts)

[VerificationException: Operation could destabilize the runtime.]

Services

ANTS runs as two services

1. ANTS Memory Profiler 4 Service
2. ANTS Performance Profiler 4 Service

Change the accounts on these two services to the same accounts used in the web application we identified above.

If this is the first time the account has been associated with a service it will be automatically granted the ‘Log On As A Service’ right.

Local Security Policy

From the Administrators menu choose Local Security Policy and grant the application pool account the following rights

1. Act as Part of the Operating System
2. Impersonate Client after authentication

Open a command prompt and run gpupdate /force.

Profiling your code

Your environment is now configured, we can start profiling.  As this is SharePoint I tend to just hit the bit i’m planning to profile first to make sure this is working before I start profiling.  Many hours can be lost trying to solve a configuration issue with a profiler that is not at fault.

Start Red Gate ANTS Performance Profiler.  It knows we’re running on a system a system that has UAC, it offers to make our life easy and always run as an Administrator.  I normally click yes here.

ANTS loads the Profile Settings window is automatically.

There are only a few things to set.

1. Choose the ASP.Net web application (hosted in IIS)
2. The URL to the SharePoint site
3. Use the original port

Further profiling counters can be set, but we just want to get our first run going for now.  Click on Start Profiling.  (In this example I have added a long running process to show how ANTS can identify this as a hotspot).

Profile Results

Profiling will start and a new browser will open at the location specified.  As this is SharePoint the profiler restarts the service so you will always have a small delay while SharePoint is JIT compiled. The profile timeline will show this impact on the processor.

Once the processor has levelled out its time to start profiling your code.   Navigate and interact with the bit of the application you are interested in.  If it is a  web part or navigation load you are interested in then you can refresh the page.

Once you are happy you have performed enough tasks,  I recommend keeping these focused, stop the profiling. ANTS will present you with the HOT code, the bit of code that is the worst performing. You can focus the analysis on any part of the profile graph to show poor performing code around a specific spike in the processor or memory.

In the example above I am focused on the code that runs (including my long running process) when I ask the magic 8 ball a question.

Viewing source code when you have enabled Line level profiling shows every line, the hit count and the % of time taken to perform that operation.

The really cool thing is you can click through the methods to follow the slow running process through the code!

Showing the method grid allows you to open up a call graph for any method to see how the processing is broken down in a dynamic tree view.  This really is just brilliant!  The image below does not do it justice you really do need to see this for yourself to appreciate it.

Conclusion

The long running process I added for the demonstration was picked up and allowed me to drill down into the lines of code that were the slowest.  It even showed how the internal calls within the base classes performed.

Red Gate ANTS Profiler is easy to setup and proves to be an invaluable tool in your software development tool kit.  I suspect profiling will not be something that you do every day, however with the speed and ease of use I would like to think that you will consider the inclusion of profiling a must have rather than a nice to have.