Agile Sharepoint development by 21apps and Andrew Woodward MVP

At the recent SharePoint Conference in Vegas I took a few minutes out to talk to Gil Zilberfeld of Typemock about what 21apps is doing with SharePoint and what I see as the next steps in the community regarding SharePoint development.

Looking at what areas I see as being a focus in the SharePoint development space,  how I will continue to push TDD but will also, now that people are starting to talk about good SharePoint development practices, start to look at the wide picture – looking at how we complement the Unit Tests with integration tests,  looking at ways to automate use acceptance tests and generally looking at ways to make the code better so that testers can focus on the scenarios and complex tests rather than dealing with the ’stupid bugs’.

Original post on Typemock Blog

Since attending Roy Osherove’s TDD Master Class I have been doing my daily TDD Kata, I will admit I have missed some days but generally I have been keeping up with it and really seeing the benefit.

Here I talked about what a Kata is and how it compares to Kumon Maths and I also took the opportunity to practice my reviewing techniques by looking at the best and reviewing Uncle Bobs prime factors TDD Kata.

I had promised to provide a web cast of the TDD SharePoint session that I did at the Best Practices conference, but I feel now that I need to really improve my keyboard mastery before taking on that one.  Instead I have taken the opportunity to record my current TDD Kata which is based on Roy Osheroves String Calculator.  I have limited the steps I have done in this recording as I think 30 minutes would get a little boring.

String Calculator

1. Create a simple String calculator with a method int Add(string numbers)
   1. The method can take 0, 1 or 2 numbers, and will return their sum (for an empty string it will return 0) for example “” or “1″ or “1,2″
   2. Start with the simplest test case of an empty string and move to 1 and two numbers
   3. Remember to solve things as simply as possible so that you force yourself to write tests you did not think about
   4. Remember to refactor after each passing test
2. Allow the Add method to handle an unknown amount of numbers
3. Allow the Add method to handle new lines between numbers (instead of commas).
   1. the following input is ok:  “1\n2,3″  (will equal 6)
   2. the following input is NOT ok:  “1,\n” 
   3. Make sure you only test for correct inputs. there is no need to test for invalid inputs for these katas
4. Allow the Add method to handle a different delimiter:
   1. to change a delimiter, the beginning of the string will contain a separate line that looks like this:   “//[delimiter]\n[numbers.]” for example “//;\n1;2″ should return three where the default delimiter is ‘;’ .
   2. the first line is optional. all existing scenarios should still be supported

The idea behind TDD is to work through one thing at a time,  so in the example we start with 0 numbers (or empty string) and progress through each requirement continuing to adopt the Red, Green, Refactor approach – hopefully this will be clear from the video.

The Kata is not about knowing how to solve this problem, it is about knowing how to do things quickly and also knowing how to approach problems using TDD. 

In this example,  as in my day job, I am using ReSharper to aid refactoring,  NUnit 2.5 for my tests as this supports TestCase and TestDriven.Net to run the tests.

I have setup some live templates for the Test method and common Assert.AreEquals statements and have added a Keyboard shortcut to Re-Run the tests in TestDriven.Net (I use Alt + R).

Hope you enjoy the video, I have left the sound off - if you perfer some nice background music I can edit and add some

Would love to hear your thoughts on this.

TDD Kata – Calculator Part 1 by AndrewWoody from Andrew Woodward on Vimeo.

You can download the WMV video to watch locally from the Vimeo site.

The numbers are in and I think this small sample really gives a good view as to the way people are approaching their SharePoint development projects.

Numbers taken from a one week poll asking people which build servers they used on their SharePoint projects.

It’s very clear from this that people are in one of 3 camps of almost identical size

• Those using Team Foundation Server
• Those that user another non Microsoft tool
• Those that don’t do anything

Team Foundation Server

Installing and using, or developing against Microsoft SharePoint Server requires a fairly big commitment to the Microsoft technology stack, so the domination of Team Foundation Server is probably to be expected.

• If your a big corporate user of SharePoint you most likely have a deal that includes TFS licenses
• You have likely committed to Microsoft as the platform of choice across the board
• If your a Microsoft Partner delivering products and solutions you will have a big investment in Microsoft licenses, and also a duty to adopt the MS recommended path
• TFS is a good solution,  and it’s getting better in 2010

Non Microsoft Tools

I was surprised by the mix of Non Microsoft tools,  having looked at what you get for free and the costs for licensing beyond that I would have expected Team City to have a bigger following.  Perhaps the reason for not being higher is jetBrains being thought of a Java based?

I bundled CruiseControl and Cruise together,  in hindsight I should have split them as they are totally different products.   I will make the assumption that most are using CruiseControl as this was really one of the first decent automated build tools to target the .Net platform.  People have got to know it an have scripted automation for their non SharePoint projects already.

Final Builder I have always liked the look of;  I really do find the whole raw XML editing that we have to do in any build automation a bit like developing against SharePoint,  it’s ok but the tools really should be better than this.   Final Builder has a great product, but the cost per user is perhaps the barrier here.

Other

Some people had build servers but didn’t use it for SharePoint – would really count those as None.

Others listed the source repository (SVN) or scripting language (msbuild, nant) so I assume that the build is done manually using batch files rather than via a dedicated build server.

Also someone listed Hudson,  something I had personally never heard of, but will bundle with the Non Microsoft Tools.

None, we don’t have a build server

This was probably the biggest surprise to me,  and perhaps not the way you might think.  I was actually surprised that only 35% of people indicated that they did not have any form of build server for their SharePoint projects.  

Based on my years of doing consultancy gigs I found that the majority of companies that I went into to do work didn’t have anything even close to an automated build server.  Heck some don’t even have source control for the code!

So I can take a couple of things from this

• People who don’t were too embarrassed to answer
• SharePoint development has moved on significantly and most teams now have proper build servers
• The readers of my blog are not representative of the wider SharePoint user base

Conclusion

There are still a lot of SharePoint teams out there that have yet to get the benefit from automating their builds,  who will undoubtedly be spending many hours manually putting things together the night before Go-Live.  

For these people, and those moving into SharePoint for the first time, there is a need to provide more information on why investing time in getting your automated build in from the start can make huge savings over the life of a project.  Providing walkthroughs on how to do this and some of the challenges you will encounter along the way.

Based on the feedback I could look at doing TFS first,  but I know that for teams with nothing setup yet this is going to be a big stumbling block.  Trying to convince you boss to roll out Team Foundation Server just so you can automate your SharePoint build is not an easy argument to have.   I will instead look at the options allowing you to get up and running quickly and more importantly without needing to get budget for licenses. 

I will start with Cruise,  and I will be dog fooding this for real on Project Aberdovey.  

Thanks to everyone who took the time to vote,  I would love to hear your own take on what the numbers mean to you and your experience with automating builds in SharePoint.

I am looking to provide some guidance on doing Continuous Integration (CI) on SharePoint projects,  looking at the concepts behind CI but also looking to provide examples that can help people get the most out of CI on a SharePoint project.   Having used a number of technologies in the past I thought it would be useful to get an idea of the which technologies people are using, if any. 

Please take a few seconds to tell me what you have used, and have a look at what others are doing.  

I have made the selections mutiple choice as I know people, especially consultants, have worked with more than one platform.  And if you haved used something not listed please add it in the other box.  

Also if you have any specific areas that you would like to see covered please add a comment.

Thanks for taking part

If you’ve read my blog you’ll know that I am a fan of Test Driven Development and an avid user of Typemock Isolator, specifically for the hard to test SharePoint code.  

If you’re not sure of the benefits of doing TDD or need to get some justification from you boss as to how quickly he could see a return on investment, have a read of this empirical evidence.

Roy Osherove is giving an hands-on TDD Masterclass in the UK, September 21-25. Roy is author of “The Art of Unit Testing” (http://www.artofunittesting.com/), a leading tdd & unit testing book; he maintains a blog at http://iserializable.com (which amoung other things has critiqued tests written by Microsoft for asp.net MVC – check out the testreviews category) and has recently been on the Scott Hanselman podcast (http://bit.ly/psgYO) where he educated Scott on best practices in Unit Testing techniques. For a further insight into Roy’s style, be sure to also check out Roy’s talk at the recent Norwegian Developer’s Conference (http://bit.ly/NuJVa). 

Full Details here: http://bbits.co.uk/tddmasterclass

bbits are holding a raffle for a free ticket for the event. To be eligible to win the ticket (worth £2395!) you MUST paste this text, including all links, into your blog and email Ian@bbits.co.uk with the url to the blog entry.  The draw will be made on September 1st and the winner informed by email and on bbits.co.uk/blog

How and I forgot to say.   I will be there as well so we can look to do a SharePint TDD edition.

OK the title does sound like a bad attempt at a pickup line, but I really mean we should talk.  I was honoured to have been asked back to talk at the Best Practices Conference in August which is my reason for being in Washington DC, but that is not the reason we should talk.

Have you implemented SharePoint? If you have, how did it go? How is it being adopted in your business?

How would you like to have four leading SharePoint thought leaders in
Organizational Change, Project Management, Requirements Development and Agile Methodology from around the world mentoring you to make sure you can deliver SharePoint success?

Not wishing to pass up on an opportunity to get together to talk SharePoint for a few days we, Andrew, Dux, Paul and Ruven, thought it would be an ideal opportunity to be a bit more productively and have set up a 3 day mentoring workshop running from August 19-21st in Reston, VA, so that we could share our experiences and knowledge in a small focused workshop.

Dux has posted full details of the workshop and why this one off event really should not be missed.

Following on from my rant response yesterday about the more negative views from the SharePoint community my reading this morning was so much more positive.

Unit Testing Workflows

In my response to a question from Aaron Weiker I said that I would like to see much more guidance and investigation into the way we approach Unit Testing workflows in SharePoint,  so it was great to find Richard Fennell is doing a lot of work in this area.  I have been working with Typemock Isolator, CThru and SilverUnit testing recently (post coming) and was very interested to see that Richard had picked up this whilst looking for a possible solution to the challenges of Unit Testing SharePoint workflows; something I have to confess I had not even considered.

I’m looking forward to seeing where Richard gets with this and also the use of Fit/Fitness, although I’m personally not a big fan of the way tables are used to define the tests.

Integration Testing SharePoint from MSTest

Having been a bit behind with my reviews of the latest Codeplex P&P project I missed the discussion with Francis on how they found a use for Typemock Isolator to do Integration tests.

The code is really very very simple,  many will be glad to hear, and deals with the places where your code makes a call to SPContext.Current or SPFarm.Local.

The simple ideas really are the best ones.

As you can see,  a few small steps for testing but big steps for SharePoint testing and a demonstration that in general the SharePoint community is committed to improving the way solutions are developed.

Just read a post by Sahil Malik where he gives his reasons for saying

TDD + SharePoint 2007? Well – screw it! Not worth it IMO.

The reason for his statement is that he sees the actual C# code developed as being a very small % of the total SharePoint project and the effort to do this as not worth the return when you consider the project costs overall.

The problem is, like so many people, he is getting TDD mixed up with doing good development.  He dismisses TDD because it is harder to do in SharePoint because of the API.  It’s not TDD that makes it harder,  it being able to isolate the API and design testable code that is harder.

If you listened to the SharePoint Podshow I did with Eric Shupps we were in agreement that what actually mattered was that you developed good quality code,  that you had the processes in place to enable code reviews, unit testing, integration testing, and the like, that all go together to ensure the code you do develop for your SharePoint solutions is Enterprise ready.

I hold my hands up, I do talks on TDD and SharePoint.  I don’t do this because I think that everyone should do this on day one,  I do it because people have used SharePoint as an excuse to not doing quality development.  The comment ‘we can’t do unit testing because we are using SharePoint’ has been common among development teams,  other just don’t do unit testing anyway so it never even cropped up.   So if I can demonstrate the you can go all the way and actually code your solution using a technique like TDD then this argument no longer holds water.

What the question should be

Quality Development and SharePoint, is it worth doing?

And the answer to that is a big Hell Yes!

And if your happen to use TDD to write some of that quality code, then that’s good too.

The key thing to remember is that you need to make a judgement call about the value; in Sahil’s defence he did mention this.  What I can’t take is that he adopts the ’screw it’ stance before he even starts. This to me is a much of a problem as the people at the other end of the spectrum that think you can ONLY code using TDD.

Scrum as an agile approach is great, it allows teams to get up to speed quickly with agile and also enable the business to understand what being agile means.  The problem is that people often just do Scrum.  Doing Scrum is not enough,  it is a great place to start and will help you with your projects but your team still need to get the engineering disciplines in place, they still need to work on doing the right thing the right way and with level of quality demanded by the business.

Jack Milunsky posted a nice article that talks about why Measuring Velocity is not enough.

Like burndown charts, velocity is just another metric which the team can use to reach what I believe is the ultimate goal – sustainable throughput. Velocity in my opinion, is not a metric for determining productivity.

What I really liked, and is something that I am going to be covering a lot more over the coming months is the conclusion

.in order to enhance productivity there are many things one can do:

* You have to get the requirements down pat. The best way to do this is to work with the customer to define the user stories and get their intent up-front.
* Ensure you are capturing the acceptance test criteria from the customer up-front. This helps solve the greatest risk of all – getting the wrong product.
* Define your definition of "done" up-front so that there’s no argument at the end.
* Ensure you build quality in right from the start.

The conclusion can be split into two areas

1. Understanding the Requirements
2. Getting your process right

Getting your process right is something the development teams control,  and there is a lot of information out there today about techniques like Test Driven Development, Continuous Integration and the like that all focus on the development process and quality.  

Understanding the the requirements on the other hand and getting the business to provide you with valid and testable user stories is an area of agile development that I feel has been neglected.  There are great books that have been written on how to write user stories and how to ensure they are testable.   There are other books aimed at the business analysts about how to gather requirements.  However these have never come together, agile development teams don’t look much be the user stories. 

This pushing back beyond the user stories and into the way requirements are gathered and the business is understood is something I have been working on for a while.  And with this push back beyond the user story it is clear the way to get business and customer buy in to the agile development approach is to really change the way businesses approach the complex problems they are trying to solve.

I have some really exciting things to announce over the coming weeks that will really take this discussion forward, working with some of the most visionary guys in the field.

Rest assured there will be a very big SharePoint focus.

I recently wrote about my experience of doing agile development in a team of one,  where I questioned the idea that ‘If your the only member of the team is it really worth going through the same agile development process that you would in a team of eight?’

One interesting test for the project was that it was not going to be developed in one go,  the days spent on this project would need to fit around other work and I want to see how using agile techniques could help to keep the momentum and also retain the knowledge from the previous activities in a format that would be easy to pick up.

Sprint 1 – Day 1 was 15th May 2009

Sprint 1 – Day 2 was 4th June 2009 a full 20 days gap!

Daily Scrum

Always start the day with a scrum and ensure that you come prepared.  This meant that I needed to be able to answer the following questions

• What did you achieve yesterday?
• What are you going to achieve today?
• Any blockers preventing you from doing this?

Take a minute,  try and think back 20 days and remember exactly what you had achieved?   Difficult isn’t it!  This is where using a process really helped,  I started by looking at the my white board to see where I left the project.

Interestingly I had not ‘Done’ anything (refer to Day 1 for the definition of Done and how I structure my white board), my Burndown for Day 1 was looking very sad so I won’t even show it.  What I could however see was that I had completed the work on one story,  the second story was in progress and that I had not started the 3rd.  The observant will also notice I have a couple of new stories in the backlog.

If I had not been working on the project for the past 20 days how come there are new stories?

Adding to the Backlog

One of the tenets of being agile is that you work at a sustainable pace,  you don’t do more work than is possible and you allow your team to go home, see the family and have a beer or two.  What happens when you team is having that beer?  They think about work, and have ideas about ways to do things or new requirements and then write these down as new stories.  The same is true for the business, whilst the team is busy working on the current iteration the business is thinking about what they need, reacting to change and then feeding this into the backlog as new stories.  They may even take away some stories that are no longer needed.

What does this mean?

Well for Sprint 1 nothing,  we have agreed a short period of work where we can focus on the deliverables.   The new stories will be considered for Sprint 2 and for any removed stories we will have the warm feeling that we had not spent too long on in big up front design tasks and documentation.

The Rest of Day 2

After the scrum I reviewed the solution I had put together,  and compared this to the stories.  At the end of Day 1 I had a WSP  and some features that meant I could see what I had done and actually use it.  The ability to use the solution very early in the process is another of the big benefits to being agile.  It allows people to see it for real, to make comment and also to make changes if it is not how they envisaged it.

Needless to say I made some changes,  there were bits that I had added that on reflection were not required,  I removed the extra items and did some refactoring of the solution to make it more compact and understandable.

The rest of the day went pretty well,  and at the end of Sprint 1 I had not actually written any C# code or created any Unit Tests.  The solution at the end of the sprint I had completed the three stories I had planned.  I have to confess that I did do a few extra hours work at the end of this day (Sprint) in order to complete the stories. This can be a common scenario where the a stress levels and work will increase towards the end of the Sprint.  The same if true of Waterfall projects but at the end of these projects there is just so much more that is expected to be done the stress levels go off the scale.

The final tasks for the end of Sprint 1 was to do a “Show and Tell” to the business.  To make this more real,  and to be able to provide a summary once the project is released I record a Screencast demonstrating what had been done.   It may sound silly doing the screen cast or doing a show and tell to yourself, however what it does do is make you bring together everything and ensure that what you said was done, actually was done.

Sprint 1 Retrospective

In order to improve our processes, to produce better quality solutions and to make ourselves more productive we need to look at what happened in the Sprint, the good things as well as the challenging ones and also look for ways to improve.

Things that went well

• Adopted great agile techniques
• Was able to pickup after a break
• Good use of tools like WSPBuilder and hosted source control

Things to improve on

• Testing was manual – need more automation of integration and acceptance tests
• Testing was on single machine – need to test on multi server farm configuration
• Project break was too long – final delivery will be too late at current pace

If you want to learn how to be better at retrospectives I recommend getting a copy of Agile Retrospectives, Making Good Teams Great by Esther Derby and Diana Larsen.

Summary

Based on the experience so far I can say that adopting agile techniques in a team of one really is something that everyone should do.   I agree just cranking out code some developers could be a fair bit more productive up front than I have been. However if the project takes off and you need to bring in more help or you have a long break the project will eventually fail without these process in place.

And the biggest benefit to doing this is that you learn the benefits of being agile,  you understand why it works and you will be able to add this to your CV and Life experience and go on to become a much better developer.  You will be able to challenge the traditional practices in your organisation based on experience and not just hearsay.