Agile Sharepoint development by 21apps and Andrew Woodward MVP

At the recent SharePoint Conference in Vegas I took a few minutes out to talk to Gil Zilberfeld of Typemock about what 21apps is doing with SharePoint and what I see as the next steps in the community regarding SharePoint development.

Looking at what areas I see as being a focus in the SharePoint development space,  how I will continue to push TDD but will also, now that people are starting to talk about good SharePoint development practices, start to look at the wide picture – looking at how we complement the Unit Tests with integration tests,  looking at ways to automate use acceptance tests and generally looking at ways to make the code better so that testers can focus on the scenarios and complex tests rather than dealing with the ’stupid bugs’.

Original post on Typemock Blog

Since attending Roy Osherove’s TDD Master Class I have been doing my daily TDD Kata, I will admit I have missed some days but generally I have been keeping up with it and really seeing the benefit.

Here I talked about what a Kata is and how it compares to Kumon Maths and I also took the opportunity to practice my reviewing techniques by looking at the best and reviewing Uncle Bobs prime factors TDD Kata.

I had promised to provide a web cast of the TDD SharePoint session that I did at the Best Practices conference, but I feel now that I need to really improve my keyboard mastery before taking on that one.  Instead I have taken the opportunity to record my current TDD Kata which is based on Roy Osheroves String Calculator.  I have limited the steps I have done in this recording as I think 30 minutes would get a little boring.

String Calculator

1. Create a simple String calculator with a method int Add(string numbers)
   1. The method can take 0, 1 or 2 numbers, and will return their sum (for an empty string it will return 0) for example “” or “1″ or “1,2″
   2. Start with the simplest test case of an empty string and move to 1 and two numbers
   3. Remember to solve things as simply as possible so that you force yourself to write tests you did not think about
   4. Remember to refactor after each passing test
2. Allow the Add method to handle an unknown amount of numbers
3. Allow the Add method to handle new lines between numbers (instead of commas).
   1. the following input is ok:  “1\n2,3″  (will equal 6)
   2. the following input is NOT ok:  “1,\n” 
   3. Make sure you only test for correct inputs. there is no need to test for invalid inputs for these katas
4. Allow the Add method to handle a different delimiter:
   1. to change a delimiter, the beginning of the string will contain a separate line that looks like this:   “//[delimiter]\n[numbers.]” for example “//;\n1;2″ should return three where the default delimiter is ‘;’ .
   2. the first line is optional. all existing scenarios should still be supported

The idea behind TDD is to work through one thing at a time,  so in the example we start with 0 numbers (or empty string) and progress through each requirement continuing to adopt the Red, Green, Refactor approach – hopefully this will be clear from the video.

The Kata is not about knowing how to solve this problem, it is about knowing how to do things quickly and also knowing how to approach problems using TDD. 

In this example,  as in my day job, I am using ReSharper to aid refactoring,  NUnit 2.5 for my tests as this supports TestCase and TestDriven.Net to run the tests.

I have setup some live templates for the Test method and common Assert.AreEquals statements and have added a Keyboard shortcut to Re-Run the tests in TestDriven.Net (I use Alt + R).

Hope you enjoy the video, I have left the sound off - if you perfer some nice background music I can edit and add some

Would love to hear your thoughts on this.

TDD Kata – Calculator Part 1 by AndrewWoody from Andrew Woodward on Vimeo.

You can download the WMV video to watch locally from the Vimeo site.

I am looking to provide some guidance on doing Continuous Integration (CI) on SharePoint projects,  looking at the concepts behind CI but also looking to provide examples that can help people get the most out of CI on a SharePoint project.   Having used a number of technologies in the past I thought it would be useful to get an idea of the which technologies people are using, if any. 

Please take a few seconds to tell me what you have used, and have a look at what others are doing.  

I have made the selections mutiple choice as I know people, especially consultants, have worked with more than one platform.  And if you haved used something not listed please add it in the other box.  

Also if you have any specific areas that you would like to see covered please add a comment.

Thanks for taking part

For almost my entire career I have been involved in software development, and probably uniquely more often than not in an environment or role where I have needed to look at process improvement. This started with my first role where I had to ensure development adhered to ISO9001 standards, through to today where I work with teams to help them adopt agile techniques and continuous improvement.

Along the way almost everyone I talked to understands the basic idea that the earlier in the cycle you find a defect the cheaper and easier it is to fix, and from this most (if not all) agree that Unit Testing is one of the most cost effective ways to catch these defects. The problem is, it appears, developers are lazy; they understand they should do it they just never get around to it – ‘perhaps on the next project’.

As part of my own development I like to work with, be trained by or just hang with people that I see as having reached a higher level of knowledge and skill than me in a particular field. You could say a greater Mastery of the subject. One area that I am very passionate about, many may have seen some great interviews on the subject, is Test Driven Development (TDD). TDD is more than just doing Unit Testing; it is a technique that once you understand and are willing to invest time in helps you to become a better developer.

I have been doing TDD, although not on every project (sadly), for about 2 years and have shared the knowledge I have gained through white papers, blogs and talking at conferences. In trying to lead the way in my area of expertise, SharePoint, I felt that I was missing something. I had yet to reach a place where I felt that I had mastered the art, where I had moved into the phase of challenging myself to do more than just the practice of TDD.

As part of my work with Typemock, the only solution of working with SharePoint’s sealed API, I found that these guys really did get it, they had the battle scars and were practicing what they preached, and they were challenging themselves to do it different, better. Roy Osherove is the Chief Architect at Typemock and I have seen him speak at conferences in a way that was engaging and thought provoking. The opportunity therefore to be able to spend 5 days with Roy doing a TDD Master Class was one that I could not miss.

I’m not going to do the, on day 1 we did this, on day two we did this, as I think that Sara (fellow attendee) has covered this quite well in her post. Instead I will talk about how the course was much more than a lesson in Test Driven Development; we paired up to cover all of the practices looking at the basics of unit testing frameworks (Nunit, MSTest), understanding isolation frameworks (Moq, Rhino Mocks, Typemock Isolator) and how they all look to solve the same issues whilst using slightly different syntax. We looked at how the techniques around TDD have evolved over time, and the how the tools have really moved on significantly in recent years making the barrier to entry and the ability to create readable and maintainable code possible. We looked at techniques, like the daily Kata, that will help perfect the practices, and discussed the ideas of Shu -Ha -Ri where the student moves from the fundamental techniques, to finding new ways and challenging tradition and then onto surpass the teacher and ultimate mastery.

I liked the way Roy questioned the idea that the lazy coder was in fact the one looking to do it as easily as possible, the one who knew all of the keyboard short cuts, the one with all the Live Templates in ReSharper. The Lazy code is really the one who gets the job done well, with the least amount of effort – unlike the average developer, who I initially thought of as lazy, who just doesn’t see development as a craft but more a way to pay the bills.

My view that learning from someone, who is at a higher level of knowledge and understanding has been reaffirmed. The same principles should apply to you, when you look at training and work, ensure that the person teaching you is someone you admire, someone who you feel has the level of knowledge and experience that you aspire to.

Having attended Roy’s TDD Master Class I can say that my knowledge has been enhanced, the core values have been reaffirmed and I am now in a much better place with regards my own capability, but more importantly my own ability to help others.

So the question is should you attend Roy’s TDD Master Class?

If you want to really Master the Art of Test Driven Development then definitely.

Before last week, I attended Roy Osherove’s TDD Masterclass,  I do not think I would have had the “you know what’s” to be able to put up an alternative solution to one of Uncle Bobs (aka Robert Martin) examples of doing Test Driven Development,  but here I am publishing the the solution I arrived at and suggesting that my solution better servers the values of Readability, Maintainability and trustworthiness.

In Uncle Bobs defence his post is dated Apr 2006 and the art of Test Driven Development has moved on leaps and bounds in the past few years.

The Problem

Write a class named “PrimeFactors” that has one static method: generate.

The generate method takes an integer argument and returns a List<Integer>. That list contains the prime factors in numerical sequence.

To see the details and how it was previously solved.

My Solution

Although the original is coded in Java and mine in C#, there is enough similarities for you to be able to understand both – weird that really

The Tests

The Production Code

I understand that showing the final solution does not prove the value of TDD, I will be posting a web cast of this process to really show how TDD helps you code to evolve and also how I adhere to the Red, Green, Refactor steps to achieve readable, maintable and trustworthy code.

Where do I think mine is better?

The Tests

The tests all adopt the naming convention Method_Scenario_Behaviour

The Act and Assert are kept separate for readability.

I have not used any logic in my tests, unlike the list()method used in Uncle Bobs.

The Production Code

No magic numbers, I have extracted well named constants.

Refactored the logic in ther generate method into more readable methods. For example line 14.

Is much easier to understand than the line

I don’t like the way the while loops were refactored into for loops to reduce the code. If you look closely at the for loop above you will see that it does not follow the normal convention where the evaluation part is related to the incremental part.  This I feel makes it easy to misread and hard to read if you don’t miss it.

Overall I think that the my code, although longer than Uncle Bobs is much easier to read and therefore easier to maintain.

I would love to hear feedback on the code, perhaps I have a couple of refactoring’s that could improve it still further.

Last week I had the pleasure of attending Roy Osherove’s Test Driven Development Masterclass,  I will post in a few days more on this but can say this is one of the best training courses I have ever attended.

During the course I got to learn about the Kata technique, something I had heard about but had never investigated.

Kata is a Japanese word describing detailed choreographed patterns of movements practiced either solo or in pairs. most commonly known for the presence in the martial arts.

The theory is that if you practice a common technique regularly (every day) over time the actions will become natural and automatic, you will have muscle memory.  During the course Roy introduced the idea of coding a reasonably detailed solution, using all of the good TDD practices (Red, Green, Refactor), every day so that the techniques used become second nature, the act of coding something using TDD is natural.

Over the course of the week the class went from being unable to code half of the problem in 30 minutes, to being able to complete the solution.  The obvious productivity’s aside, I will discuss this in a future post, it was clear that every developer in the class was able to master a new technique.   In fact I have tweeted with some of the class members who were doing the same code Kata on Saturday morning.

So what does Jack do?

Although he did join me in my Sunday morning Kata I am not saying that Jack has been doing these coding Katas for years. No, Jack has been doing Kumon mathematics since he was 4 years old. 

Kumon is a math and reading enrichment program. Students do not work together as a class, but progress through the curriculum at their own pace, moving on to the next level when they have achieved mastery of the previous level. Mastery is defined as speed and accuracy.  They only progress when they have mastered the skills through practice and repetition.

Kumon sheets (numbers of questions) are completed daily, they build up the students skills in mental arithmetic and techniques by repeating them often.   Jack, like all new students, started at a level that is below their current ability but through which they can quickly progress.  Jack started at level 6A and moved quickly through the numbers bonds and now has an amazing ability to do mental maths.  Although challenging at times Jack has continued with his daily Kumon (KATA) and had reached the G-League before starting high school, an amazing achievement.

So I can now say that Jack and I will be doing our daily Kata together,  perhaps aiming to master a different art, but taking a daily step closer to achieving that mastery.

Kata is definitely something that will become a way of life for people at 21apps, the value is real and measurable, the personal satisfaction is infectious.

Just read a post by Sahil Malik where he gives his reasons for saying

TDD + SharePoint 2007? Well – screw it! Not worth it IMO.

The reason for his statement is that he sees the actual C# code developed as being a very small % of the total SharePoint project and the effort to do this as not worth the return when you consider the project costs overall.

The problem is, like so many people, he is getting TDD mixed up with doing good development.  He dismisses TDD because it is harder to do in SharePoint because of the API.  It’s not TDD that makes it harder,  it being able to isolate the API and design testable code that is harder.

If you listened to the SharePoint Podshow I did with Eric Shupps we were in agreement that what actually mattered was that you developed good quality code,  that you had the processes in place to enable code reviews, unit testing, integration testing, and the like, that all go together to ensure the code you do develop for your SharePoint solutions is Enterprise ready.

I hold my hands up, I do talks on TDD and SharePoint.  I don’t do this because I think that everyone should do this on day one,  I do it because people have used SharePoint as an excuse to not doing quality development.  The comment ‘we can’t do unit testing because we are using SharePoint’ has been common among development teams,  other just don’t do unit testing anyway so it never even cropped up.   So if I can demonstrate the you can go all the way and actually code your solution using a technique like TDD then this argument no longer holds water.

What the question should be

Quality Development and SharePoint, is it worth doing?

And the answer to that is a big Hell Yes!

And if your happen to use TDD to write some of that quality code, then that’s good too.

The key thing to remember is that you need to make a judgement call about the value; in Sahil’s defence he did mention this.  What I can’t take is that he adopts the ’screw it’ stance before he even starts. This to me is a much of a problem as the people at the other end of the spectrum that think you can ONLY code using TDD.

If your the only member of the team is it really worth going through the same agile development process that you would in a team of eight?

Isn’t it quicker to just crack open Visual Studio and start coding?   After all your the customer so you know what you want right?   You not going to have to deal with any of the social complexities that conspire against you. You’re going have the hottest of communication with the dev team,  the dev and test roles will have an interment knowledge and the scrum master won’t really have to do anything as there is not going to be any blockers on this project.

Right?

So I started down this path and before I knew it I had some weird project created with bits of code and SharePoint artefacts but not really seeming to make any progress.   I was easily distracted by other things and didn’t really have a feel for how long it would take and when I could say I was done.

STOP!

What was I doing!  I spend my time trying to help people develop better, to take small steps and sometimes big steps to help produce better software..   why was I not practicing what I preached?

I stopped!

Deep breath, deleted all the rubbish I can previously created and got back to basics.

The Team

I have a team,  the fact that some (in this case all) of the roles were done by the same person was ok; that happens often in agile teams.

Design

Yes, you heard me right here.  Eric Shupps would be proud of me .   I’m not going to jump into Visual Studio and crank out code starting [TestFixture] I need to do some design work first, even though this application is going to be small I still need to put in the time on the upfront design.  I need to think through the solution at a high level.

Here I used the opportunity to try out the great Balsamiq application,  which I stumped up the $79 license fee for and download the desktop version.

I’ve deliberately blurred the image but you get the idea of what the mock ups look like, and these take only a matter of minutes to produce.  Definately something you should consider if your currently trying to do this in Visio or heaven forbid cranking out HTML mockups.

The few UI based mockups actually helped me to understand how the application would work for the end user, which in turn provided a clear vision for the way the solution would be developed.

User Stories

I didn’t need to do any remote collaboration on the User Stories so I was able to get back to basics and use one of the white boards in my office as the task board.   I wrote the user stories out on post-it notes and added them to the board along with rough estimates for each one.

Here you can see I’m using a very common step of status on my task board

Backlog - This is where I add all stories that came out of the design,  they are all user focused with nothing specific about the implementation.  No stories like ‘Use jQuery to make it nice’  these are user stories.  If I think of something while i’m coding or testing, or even having a beer I can just write it to a post-it and stick it on the backlog.

Sprint – I like Scrum so adopt a Scrum Esq. style to my process.  The sprint will be where I put the stories I plan to do next.  The amount is based on the estimate and also the sprint length.   In this project I’m doing stupily short sprints of 2 days each.

In Progress – When I pick off a story to work on it goes in here,  if you’ve ever looked at the Kanban approaches your looking to keep this ‘In Progress’ to a defined level.  I like to try and move things through to ‘Done’ before picking off new stories.

Verify – When I think it’s been coded properly and dev tested I move the story here.   Verify is where I would validate the code on another environment – i.e. a non-dev box,  one with multiple web front ends.  At this point I also look at introducing some form of automation for both the build/deployment and also the acceptance testing.

Done – This is when it is really really done,  there is nothing left to do, no documentation, no tests or tweaks,  the code is production ready.

Sprint 1 – Day 1

Having selected the user stories for the sprint it was time to setup my dev project and get some of the foundations in place.   If you follow me on twitter you will know that I previously spent time producing a decent SharePoint development Virtual Machine so it was very quick for me to roll out the few things I needed to get going.   A SharePoint team site in which I could spike and test my code,  a VS2008 solution and class library project in which to start cranking out some code.

At the end of Day 1 my board looked like this:

One of the stories I felt was ready for verification and another I was making good progress on.   At this point I had nothing ‘Done’ my burndown in the morning would look a bit sad,  but that’s OK that is often the case at the start of a project or sprint.  I also don’t at this point have a verification setup so this is going to slow me down a bit on Day 2.

Day 1 has been good,  I have the basic project structure defined.   I was able to build a WSP with the SharePoint features I needed to complete the first story and was able to deploy and test this on my dev machine.

People that have read my blog or seen me talk at conferences and user groups know that I am very keen on doing things right; I like to see Unit Tests and better still I want to see people at least trying Test Driven Development.   However at the end of Day 1 I was yet to write a single line of C# code, neither in test form or production code.   So did I really do development on day one?

Well I think I did, only the language used was XML and a little bit of CAML and for these items I do not know of any way, that makes sense, to unit test.  I often think that the XML we define as part of SharePoint projects are more inline with doing configuration,  we are just setting up SharePoint using a specific configuration and the testing for this should naturally falls into the integration space and will be picked up during verification.   That is not to say as a developer I do not validate it,  I need to ensure that it works as expected in my environment before I can move it over to the Verify column and it’ is the job of the tester to make sure this happens.  For this project the tester (me) was adamant that it all worked as they didn’t want to have to waste time on silly errors.

Interesting Test

One thing that I will be having to do with this project is fit it around real work, the sort that helps to keep food on the table sort of work.   This means that the sprints although 2 days in length the elapsed time may be significantly longer.  I know that this will impact on the velocity as I will lose some of the momentum during these breaks, however the benefit I have from adopting a more defined process is that it will be significantly easier for me to pick up the project.   Imaging if I had to go back to the mess of code I had originally created after a weeks break!

I’m really interested to see if my agile development with a team of one really does make a significant impact on the way I develop the code, on the time it takes to complete it and on the quality of the solution created at the end.

Find out what happens on Day 2.

How many times have you added an additional parameter or bit of logic to your code because you thought it could be useful and it’s easier to add whilst your already changing the code?   If you are your introducing Future Creep!  and should stop doing it.

What does Future Creep mean?

If your from from an XP (Extreme Programming) background you would know this better as YAGNI,  You Aren’t Gonna Need It!   I actually prefer the term Future Creep I first saw on the post, beware of future creep by Jamis of 37 signal.   Everyone understands the terms Scope Creep and Feature Creep and its easy to spot these, although often ignored.  Future creep is the adding of additional code by the developer because they can see a possible future requirements.

DONT DO IT!

You may think, hey it’s only gonna take me a couple of minutes to add this additional parameter which I can see being really useful in the future.  You add the code and then move on,  from that point forward that additional code needs to be maintained, every time a developer interacts with your code they need to read it and understand it – often taking more time as that bit of the code is never actually used, which leads to uncertainty about the code.

It happened to me

The reason I wrote this blog post was an example of future creep on a recent project that had a measurable impact on time and effort. The future creep was was simple, the developer was writing a light weight logging library and decided as part of the Logging.Log() method that if the code had a HTTPContext they would also initialise the current web using SPContext.Current.   None of the code within the internal methods made use of this it was just setup and passed into the method.  This in itself does not sound to bad and for most scenarios the code did run with a valid SPContext h0wever this future creep caused a number of issues which forced developers trying work around it

1. The logging library could not be used in code that had elevated privileges as SPContext.Current throws an error if accessed under a different security context
2. The logging library could not be used where a HTTPContext existed but no SPContext, as was the case for some of the Exchange integration code

Having battled with solutions to the second scenario and getting comments about not wanting to make changes to the Logging Library because the dev was not comfortable with the way it had been implemented I challenged the original author of the code as to the requirement for SPContext.

The answer:  Future Creep

A few minutes later the extra code was refactored out, and the library could be used.

Lessons Learnt and how to prevent it?

Don’t do it,  resist the temptation, remember:

The best way to implement code quickly is to implement less of it.

The best way to have fewer bugs is to implement less code.

And adopting techniques like Test Driven Development force you down a path that doesn’t allow it.

Ever wondered if it was possible to do Test Driven Development on SharePoint projects?   Have you been reading the blog posts and white papers here at 21apps.  Got involved in the debate with Eric Shupps?

The SharePoint Pod Show team finally tracked me, Andrew Woodward, down for a series of in depth interviews late one night during the MVP Summit 2009.   The Pod Show is broken into 3 parts with Rob Foster asking about the basics of Test Driven Development and SharePoint;  Eric Shupps, the SharePoint Cowboy, asks about the value TDD provides to the Enterprise and finally Spencer Harbar asks the challenging questions about how TDD helps reduce the risks to the enterprise by adopting TDD.

Hopefully you will agree that some of the myths around TDD have been dispelled by these insightful SharePoint Pod Shows.   I encourage you all to sign up for the European Best Practices SharePoint Conference in London, 4th to 6th April 2009, where we hope to continue the discussion around this challenging topic.

Don’t forget to sign up to the legendary SharePint once again organised by the Andrew Connell.

Finally, in the words of Spencer Harbar (sorry Eric no pictures of Spence!)