Agile Sharepoint development by 21apps and Andrew Woodward MVP

The numbers are in and I think this small sample really gives a good view as to the way people are approaching their SharePoint development projects.

Numbers taken from a one week poll asking people which build servers they used on their SharePoint projects.

It’s very clear from this that people are in one of 3 camps of almost identical size

• Those using Team Foundation Server
• Those that user another non Microsoft tool
• Those that don’t do anything

Team Foundation Server

Installing and using, or developing against Microsoft SharePoint Server requires a fairly big commitment to the Microsoft technology stack, so the domination of Team Foundation Server is probably to be expected.

• If your a big corporate user of SharePoint you most likely have a deal that includes TFS licenses
• You have likely committed to Microsoft as the platform of choice across the board
• If your a Microsoft Partner delivering products and solutions you will have a big investment in Microsoft licenses, and also a duty to adopt the MS recommended path
• TFS is a good solution,  and it’s getting better in 2010

Non Microsoft Tools

I was surprised by the mix of Non Microsoft tools,  having looked at what you get for free and the costs for licensing beyond that I would have expected Team City to have a bigger following.  Perhaps the reason for not being higher is jetBrains being thought of a Java based?

I bundled CruiseControl and Cruise together,  in hindsight I should have split them as they are totally different products.   I will make the assumption that most are using CruiseControl as this was really one of the first decent automated build tools to target the .Net platform.  People have got to know it an have scripted automation for their non SharePoint projects already.

Final Builder I have always liked the look of;  I really do find the whole raw XML editing that we have to do in any build automation a bit like developing against SharePoint,  it’s ok but the tools really should be better than this.   Final Builder has a great product, but the cost per user is perhaps the barrier here.

Other

Some people had build servers but didn’t use it for SharePoint – would really count those as None.

Others listed the source repository (SVN) or scripting language (msbuild, nant) so I assume that the build is done manually using batch files rather than via a dedicated build server.

Also someone listed Hudson,  something I had personally never heard of, but will bundle with the Non Microsoft Tools.

None, we don’t have a build server

This was probably the biggest surprise to me,  and perhaps not the way you might think.  I was actually surprised that only 35% of people indicated that they did not have any form of build server for their SharePoint projects.  

Based on my years of doing consultancy gigs I found that the majority of companies that I went into to do work didn’t have anything even close to an automated build server.  Heck some don’t even have source control for the code!

So I can take a couple of things from this

• People who don’t were too embarrassed to answer
• SharePoint development has moved on significantly and most teams now have proper build servers
• The readers of my blog are not representative of the wider SharePoint user base

Conclusion

There are still a lot of SharePoint teams out there that have yet to get the benefit from automating their builds,  who will undoubtedly be spending many hours manually putting things together the night before Go-Live.  

For these people, and those moving into SharePoint for the first time, there is a need to provide more information on why investing time in getting your automated build in from the start can make huge savings over the life of a project.  Providing walkthroughs on how to do this and some of the challenges you will encounter along the way.

Based on the feedback I could look at doing TFS first,  but I know that for teams with nothing setup yet this is going to be a big stumbling block.  Trying to convince you boss to roll out Team Foundation Server just so you can automate your SharePoint build is not an easy argument to have.   I will instead look at the options allowing you to get up and running quickly and more importantly without needing to get budget for licenses. 

I will start with Cruise,  and I will be dog fooding this for real on Project Aberdovey.  

Thanks to everyone who took the time to vote,  I would love to hear your own take on what the numbers mean to you and your experience with automating builds in SharePoint.

I am looking to provide some guidance on doing Continuous Integration (CI) on SharePoint projects,  looking at the concepts behind CI but also looking to provide examples that can help people get the most out of CI on a SharePoint project.   Having used a number of technologies in the past I thought it would be useful to get an idea of the which technologies people are using, if any. 

Please take a few seconds to tell me what you have used, and have a look at what others are doing.  

I have made the selections mutiple choice as I know people, especially consultants, have worked with more than one platform.  And if you haved used something not listed please add it in the other box.  

Also if you have any specific areas that you would like to see covered please add a comment.

Thanks for taking part

OK the title does sound like a bad attempt at a pickup line, but I really mean we should talk.  I was honoured to have been asked back to talk at the Best Practices Conference in August which is my reason for being in Washington DC, but that is not the reason we should talk.

Have you implemented SharePoint? If you have, how did it go? How is it being adopted in your business?

How would you like to have four leading SharePoint thought leaders in
Organizational Change, Project Management, Requirements Development and Agile Methodology from around the world mentoring you to make sure you can deliver SharePoint success?

Not wishing to pass up on an opportunity to get together to talk SharePoint for a few days we, Andrew, Dux, Paul and Ruven, thought it would be an ideal opportunity to be a bit more productively and have set up a 3 day mentoring workshop running from August 19-21st in Reston, VA, so that we could share our experiences and knowledge in a small focused workshop.

Dux has posted full details of the workshop and why this one off event really should not be missed.

A big pat on the back for Alex Pearce and Richard Willis for organising the first every Microsoft Learning Gateway conference that took place on last week.  With such a focused topic as SharePoint in Education it was amazing too see that there were over 150 people packing out the two session rooms.

As promised I have uploaded my slide deck to Slideshare.  I have also included the option to download the original PowerPoint presentation as some of you may want to use this to discuss your own SharePoint deployments.

I do plan to do a screen cast of the presentation later in the year to provide some background for those that were unable to make it.     If you didn’t attend and you didn’t follow the torrent of tweets I did during the day you can see them here http://search.twitter.com/search?q=lgcuk

Following on from my rant response yesterday about the more negative views from the SharePoint community my reading this morning was so much more positive.

Unit Testing Workflows

In my response to a question from Aaron Weiker I said that I would like to see much more guidance and investigation into the way we approach Unit Testing workflows in SharePoint,  so it was great to find Richard Fennell is doing a lot of work in this area.  I have been working with Typemock Isolator, CThru and SilverUnit testing recently (post coming) and was very interested to see that Richard had picked up this whilst looking for a possible solution to the challenges of Unit Testing SharePoint workflows; something I have to confess I had not even considered.

I’m looking forward to seeing where Richard gets with this and also the use of Fit/Fitness, although I’m personally not a big fan of the way tables are used to define the tests.

Integration Testing SharePoint from MSTest

Having been a bit behind with my reviews of the latest Codeplex P&P project I missed the discussion with Francis on how they found a use for Typemock Isolator to do Integration tests.

The code is really very very simple,  many will be glad to hear, and deals with the places where your code makes a call to SPContext.Current or SPFarm.Local.

The simple ideas really are the best ones.

As you can see,  a few small steps for testing but big steps for SharePoint testing and a demonstration that in general the SharePoint community is committed to improving the way solutions are developed.

Just read a post by Sahil Malik where he gives his reasons for saying

TDD + SharePoint 2007? Well – screw it! Not worth it IMO.

The reason for his statement is that he sees the actual C# code developed as being a very small % of the total SharePoint project and the effort to do this as not worth the return when you consider the project costs overall.

The problem is, like so many people, he is getting TDD mixed up with doing good development.  He dismisses TDD because it is harder to do in SharePoint because of the API.  It’s not TDD that makes it harder,  it being able to isolate the API and design testable code that is harder.

If you listened to the SharePoint Podshow I did with Eric Shupps we were in agreement that what actually mattered was that you developed good quality code,  that you had the processes in place to enable code reviews, unit testing, integration testing, and the like, that all go together to ensure the code you do develop for your SharePoint solutions is Enterprise ready.

I hold my hands up, I do talks on TDD and SharePoint.  I don’t do this because I think that everyone should do this on day one,  I do it because people have used SharePoint as an excuse to not doing quality development.  The comment ‘we can’t do unit testing because we are using SharePoint’ has been common among development teams,  other just don’t do unit testing anyway so it never even cropped up.   So if I can demonstrate the you can go all the way and actually code your solution using a technique like TDD then this argument no longer holds water.

What the question should be

Quality Development and SharePoint, is it worth doing?

And the answer to that is a big Hell Yes!

And if your happen to use TDD to write some of that quality code, then that’s good too.

The key thing to remember is that you need to make a judgement call about the value; in Sahil’s defence he did mention this.  What I can’t take is that he adopts the ’screw it’ stance before he even starts. This to me is a much of a problem as the people at the other end of the spectrum that think you can ONLY code using TDD.

There has been a noticeable increase in the amount of people now blogging and tweeting about SharePoint in Education, or more accurately the Microsoft Learning Gateway platform.  Having posted a number of articles on the subject and having started the Education Special Interest group on SUGUK I thought it time I posted about some of this content.

The first an probably the most important change is the work Alex Pearce has done in rescuing what was really a failed attempt by Microsoft to promote MLG and re launch the http://www.learninggateway.net/ site and launch the Learning Gateway User Group

What is the Learning Gateway User Group?
This community site is here for you.  A place where you can share your personal experiences with the Learning Gateway.  You may just have Microsoft SharePoint as your Learning Gateway and you want to show off the work you have done, or you are a school and you want to talk about all the great ways students and teachers and working together, enriching the learners.

There are a number of bloggers who have moved their blogs to the site and we are starting to see some great posts on there.

To really kick start this drive Alex and Richard Willis (SalamanderSoft) have organised the first conference dedicated to the Microsoft Learning Gateway.

It is promising to be the best education SharePoint event this year for both technical and non-technical staff. Whether you are a Network Manager, VLE Co-ordinator or Head Teacher you will find this conference an opportunity to learn from industry recognised experts speaking about the technical best practice of the Learning Gateway and how to get your staff, pupils and parents engaged and secure. In addition you’ll be able to network with your peers and see how others are pushing forward their Gateway.

There are some great speakers lined up for the event including 5 MVPs in SharePoint and Exchange with the Keynote by Steve Beswick, Directory of Education at Microsoft.

SharePoint in Education

Mike Herrity, from Twynham School, has launched a fantastic blog where he is really sharing his experiences in working with SharePoint in Education.   Unlike myself who tends to work on the larger projects with local authorities and partners Mike is an Assistant Head Teacher so really does talk about what it’s like in the school.

http://sharepointineducation.com/

As I find more great resources I will add then to this post,  if you blog or tweet on the Microsoft Learning Gateway please add a comment below with a link.

If your the only member of the team is it really worth going through the same agile development process that you would in a team of eight?

Isn’t it quicker to just crack open Visual Studio and start coding?   After all your the customer so you know what you want right?   You not going to have to deal with any of the social complexities that conspire against you. You’re going have the hottest of communication with the dev team,  the dev and test roles will have an interment knowledge and the scrum master won’t really have to do anything as there is not going to be any blockers on this project.

Right?

So I started down this path and before I knew it I had some weird project created with bits of code and SharePoint artefacts but not really seeming to make any progress.   I was easily distracted by other things and didn’t really have a feel for how long it would take and when I could say I was done.

STOP!

What was I doing!  I spend my time trying to help people develop better, to take small steps and sometimes big steps to help produce better software..   why was I not practicing what I preached?

I stopped!

Deep breath, deleted all the rubbish I can previously created and got back to basics.

The Team

I have a team,  the fact that some (in this case all) of the roles were done by the same person was ok; that happens often in agile teams.

Design

Yes, you heard me right here.  Eric Shupps would be proud of me .   I’m not going to jump into Visual Studio and crank out code starting [TestFixture] I need to do some design work first, even though this application is going to be small I still need to put in the time on the upfront design.  I need to think through the solution at a high level.

Here I used the opportunity to try out the great Balsamiq application,  which I stumped up the $79 license fee for and download the desktop version.

I’ve deliberately blurred the image but you get the idea of what the mock ups look like, and these take only a matter of minutes to produce.  Definately something you should consider if your currently trying to do this in Visio or heaven forbid cranking out HTML mockups.

The few UI based mockups actually helped me to understand how the application would work for the end user, which in turn provided a clear vision for the way the solution would be developed.

User Stories

I didn’t need to do any remote collaboration on the User Stories so I was able to get back to basics and use one of the white boards in my office as the task board.   I wrote the user stories out on post-it notes and added them to the board along with rough estimates for each one.

Here you can see I’m using a very common step of status on my task board

Backlog - This is where I add all stories that came out of the design,  they are all user focused with nothing specific about the implementation.  No stories like ‘Use jQuery to make it nice’  these are user stories.  If I think of something while i’m coding or testing, or even having a beer I can just write it to a post-it and stick it on the backlog.

Sprint – I like Scrum so adopt a Scrum Esq. style to my process.  The sprint will be where I put the stories I plan to do next.  The amount is based on the estimate and also the sprint length.   In this project I’m doing stupily short sprints of 2 days each.

In Progress – When I pick off a story to work on it goes in here,  if you’ve ever looked at the Kanban approaches your looking to keep this ‘In Progress’ to a defined level.  I like to try and move things through to ‘Done’ before picking off new stories.

Verify – When I think it’s been coded properly and dev tested I move the story here.   Verify is where I would validate the code on another environment – i.e. a non-dev box,  one with multiple web front ends.  At this point I also look at introducing some form of automation for both the build/deployment and also the acceptance testing.

Done – This is when it is really really done,  there is nothing left to do, no documentation, no tests or tweaks,  the code is production ready.

Sprint 1 – Day 1

Having selected the user stories for the sprint it was time to setup my dev project and get some of the foundations in place.   If you follow me on twitter you will know that I previously spent time producing a decent SharePoint development Virtual Machine so it was very quick for me to roll out the few things I needed to get going.   A SharePoint team site in which I could spike and test my code,  a VS2008 solution and class library project in which to start cranking out some code.

At the end of Day 1 my board looked like this:

One of the stories I felt was ready for verification and another I was making good progress on.   At this point I had nothing ‘Done’ my burndown in the morning would look a bit sad,  but that’s OK that is often the case at the start of a project or sprint.  I also don’t at this point have a verification setup so this is going to slow me down a bit on Day 2.

Day 1 has been good,  I have the basic project structure defined.   I was able to build a WSP with the SharePoint features I needed to complete the first story and was able to deploy and test this on my dev machine.

People that have read my blog or seen me talk at conferences and user groups know that I am very keen on doing things right; I like to see Unit Tests and better still I want to see people at least trying Test Driven Development.   However at the end of Day 1 I was yet to write a single line of C# code, neither in test form or production code.   So did I really do development on day one?

Well I think I did, only the language used was XML and a little bit of CAML and for these items I do not know of any way, that makes sense, to unit test.  I often think that the XML we define as part of SharePoint projects are more inline with doing configuration,  we are just setting up SharePoint using a specific configuration and the testing for this should naturally falls into the integration space and will be picked up during verification.   That is not to say as a developer I do not validate it,  I need to ensure that it works as expected in my environment before I can move it over to the Verify column and it’ is the job of the tester to make sure this happens.  For this project the tester (me) was adamant that it all worked as they didn’t want to have to waste time on silly errors.

Interesting Test

One thing that I will be having to do with this project is fit it around real work, the sort that helps to keep food on the table sort of work.   This means that the sprints although 2 days in length the elapsed time may be significantly longer.  I know that this will impact on the velocity as I will lose some of the momentum during these breaks, however the benefit I have from adopting a more defined process is that it will be significantly easier for me to pick up the project.   Imaging if I had to go back to the mess of code I had originally created after a weeks break!

I’m really interested to see if my agile development with a team of one really does make a significant impact on the way I develop the code, on the time it takes to complete it and on the quality of the solution created at the end.

Find out what happens on Day 2.

UPDATE:  Project Aberdovey is now live and called 21SCRUM

Download a trial from www.21scrum.com

Being an advocate of SharePoint as an application platform I looked long and hard for a decent implementation of a Scrum and have always been found wanting.  I have seen simple examples where any power user could set things up but found the lack of any Burndown charts meant you lost probably the most important part of Scrum – the visibility.

I have found that VersionOne is one of the best products on the market, although it has a lot of options which tends to put off new teams.

For those that use Team Foundation Server your probably going to head towards the Scrum for Team System developed by Conchango.  I’ve not used this in anger so am unable to comment on how well it works however this solution is not really a SharePoint specific solution as it requires you to make the commitment to using Team Foundation Server.

Bil Simser did a good post on scrum tools,  although its nearly 3 years old still has some good links.   Again however the tools are not specifically SharePoint.

Why the SharePoint Obsession?

SharePoint is the collaboration tool of choice, companies have deployed it and it is being used with varying degrees of success.  People have become familiar with the how to add items to a list, how to upload documents into a document library and for the more advanced how to build engaging dashboards.

What is the biggest thing you get from adopting scrum?  ‘Visibility’

What do you need to encourage to make Scrum work?  ‘Collaboration’

SharePoint is the natural platform choice on which to build a Scrum tool.

Introducing Project Aberdovey

As with all good development teams I think it is right that your development project should have a name,  and in keeping with some big companies in the Seattle area 21apps uses place names.   Aberdovey, or as it’s spelt in Welsh Aberdyfi, has probably one of the best beaches in the UK and is located 50 miles west of the office.

Put simply, Project Aberdovey is a Scrum tool for SharePoint.

There are a few aims for the project:

1. Produce a fantastic Scrum tool built on SharePoint
2. Develop the solution using agile techniques including TDD
3. Dogfood the solution as soon as possible – what better way to develop the right solution
4. Open development – I want to give feedback on the challenges but also welcome your input

I will be looking for ways to give people visibility,  I am looking at codeplex as an option but welcome any suggestions.

Ever wondered if it was possible to do Test Driven Development on SharePoint projects?   Have you been reading the blog posts and white papers here at 21apps.  Got involved in the debate with Eric Shupps?

The SharePoint Pod Show team finally tracked me, Andrew Woodward, down for a series of in depth interviews late one night during the MVP Summit 2009.   The Pod Show is broken into 3 parts with Rob Foster asking about the basics of Test Driven Development and SharePoint;  Eric Shupps, the SharePoint Cowboy, asks about the value TDD provides to the Enterprise and finally Spencer Harbar asks the challenging questions about how TDD helps reduce the risks to the enterprise by adopting TDD.

Hopefully you will agree that some of the myths around TDD have been dispelled by these insightful SharePoint Pod Shows.   I encourage you all to sign up for the European Best Practices SharePoint Conference in London, 4th to 6th April 2009, where we hope to continue the discussion around this challenging topic.

Don’t forget to sign up to the legendary SharePint once again organised by the Andrew Connell.

Finally, in the words of Spencer Harbar (sorry Eric no pictures of Spence!)