In our last post, I ain’t got no requirements! We talked about the approach we take to help our customers focus on real business needs rather than technology. Helping them to step back from the SharePoint 2010 feature list and really understand what it is they are trying to achieve.
We talked about the great facilitation techniques we use, like Innovation Games and Dialog Mapping, to get everyone on the same page, uncover any underlying issues and explore wider aspirations. We are also doing some great work in this area about ways of working within an organisation or in the more traditional partner/customer relationship, but that’s for a future post.
In this post we are going to look at the other aspect the Outcomes stage of the 21Shift model – SharePoint 2010 business value, how do you measure it?
What we measure today
To be honest not most don’t measure a lot really! There are many reasons why an they might not measure the impact of deploying SharePoint:
- They never really had a goal or vision against which to measure
- It’s too hard
- They think they are measuring – IT has a budget they are tracking spend
Most organisations that deploy SharePoint have limited if any stated goals for the deployment, and those that do almost always look for safety of ambiguous statements and platitudes.
Does this sound familiar?
“To improve collaboration between our employees”
- What does that mean?
- How are you going to measure “improved collaboration”?
- What business value does “improved collaboration” bring?
I am sure you can have a go at answering these questions; the problem is most of you will come up with your own view. You won’t all be on the same page. Ambiguous statements are easy to agree on, you need to be able to reasonably disagree with an objective, or measure it for it to be of value.
Try this example
If you’re reading this post you are likely to have been involved with or are currently working on a SharePoint implementation. Go grab the ‘vision’ for that project (if you have one that is :)) and see if you can apply the following test.
Add your vision to the end of the sentence:
“The Montgomery Burns award for outstanding achievement in .. “
Does it sound ok?
In our example it reads like this:
“The Montgomery Burns award for outstanding achievement in improved collaboration between our employees”
If yours fits here, like the example above then I’m sorry to have to tell you that you’re dealing with a Platitude, and at risk of spending lots on money deploying SharePoint only to look back in a few years and say ‘we never planned to be here. perhaps SharePoint v.Next will be the answer’!
The Montgomery Burns award test was created by Paul Culmsee, of Seven Sigma, and is just one of the many techniques covered in his SharePoint Governance and Information Master Class.
What is measured isn’t valuable
Not all projects are completely free of measures; the problem is the measures you are likely performing are the ones with the lowest information value.
Quote Douglas W. Hubbard
The Measurement Inversion
In a business case, the economic value of measuring a variable is usually inversely proportional to how much measurement attention it usually gets.
As shown in the diagram:
Project teams are good at measuring the project costs, how much licenses will cost and what servers are needed, they also like to show how these extrapolate out over the long term in maintenance and support costs. However as the information value of the measures increase so the amount of measures decrease.
What we should be measuring
Before we can measure anything we need to look at an example of a well written objective, one that is not based on platitudes and that stands up the being asked “Why” and “How”.
Recently I’ve been making great use of the Kapitola Pathway technique that was introduced to me by Paul Culmsee. An example output from a SharePoint project might look like this.
In this example we have a stated measurable objective:
“We are proving that, through the use of collaboration tools, we can capture, use, find and protect our information and knowledge in a way that reduces duplication, increases effectiveness of our teams, and retains knowledge within the organisation.”
Using the Kapitola Pathway technique we can read to the right how we plan to achieve this objective, and in a similar way we can also read from right to left. Note, this is not complete and only used as an example to allow us to look at the measurements.
- How many measures can you get from this objective?
- What words in this objective stand out to you as something that you might want to measure?
- If we asked the question “Why” what might the wider business goals look like? Does this objective actually align?
One of the most common measures that will be present in any SharePoint deployment is that of information availability. Our objective mentions increased effectiveness, reduced duplication which you could argue sound like platitudes – however we are stating ‘We are proving that.’
We are proving that through the use of measures we can demonstrate increased effectiveness, reduced duplication.
How would we do that?
You have to keep in mind here that we are measuring something specific, we are not trying to measure organisational wide effectiveness at this level, what we are looking to do is measure how “through the use of collaboration tools to capture, use, find and protect our information we can increase our teams effectiveness..”
What things would a team do that would not be effective?
- Recreating information that already exists
- Spending time searching for information
- Making decisions without all of the information available to them, because they cannot find it
- Using inaccurate or out of date information
Based on these we could then start to think about how we would measure improvements, firstly looking at tasks we shouldn’t need to do
- Cost of effort to recreate a document
- Cost in regards time looking for a document
- Cost of making wrong decisions due to lack of information
Based on these we could start to estimate
- How long does it take on average to recreate a document
- How much time is spent searching
- How many decisions are made without all of the information being available
How do we measure these?
Before you can start to quantify things you need to understand the difference between accuracy and precision.
Example of precision
It takes 12.5 hours on average to produce a document
Example of accuracy
It takes between 6 and 20 hours on average to produce a document
Which is better?
How often have you seen project managers produce precise costs for a project based on information entered into timesheets?
How often have you actually recorded with a stop watch the precise time you spent on each task? Probably never, what you will have done is roughly apportion your time out across the various tasks to the nearest hour.
So the precise figure quoted by the Project Manager is never precise!
What we are looking for is an accurate estimate, one that you would have a 90% confidence in it being correct – to use the terminology of Douglas W. Hubbard a 90% Confidence Interval (CI).
Calibrating your estimate
How do you calibrate your estimates?
Me: How long does it take you on average to create a standard document? I’m looking for you to give me 90% confidence interval.
SharePoint User: It depends? I really don’t know how long, some documents are very short others are huge. It’s not something we record.
Me: Of course you don’t know exactly how long it takes, that is why we use a range. What would be the longest time actually spent creating a document?
SharePoint User: Do mean elapsed time?
Me: We are looking at this as way to measure the effort needed to recreate a document you can’t find so this would be about actual time spent working on the document not waiting for people.
SharePoint User: I don’t know it varies so much.
Me: Would it ever take more than 100 hours?
SharePoint User: No, not now we have moved away from doing those old monolithic documents so never more than 100 hours.
Me: Could it take more than 50 hours for the longest document?
SharePoint User: Probably yes
Me: Looking for your 90% confidence interval of the average time to recreate a document. If you consider all of the documents created could the average be more than 50 hours?
SharePoint User: I see what you mean. I would say the average is probably less than 50 hours.
Me: So your upper bound for the average would be.?
SharePoint User: Okay, it is highly unlikely that the average time to recreate a document is greater than 20 hours.
Me: Great, Now let’s consider the lower bound. How small could it be?
SharePoint User: Some documents are very simple, probably only taking 2 hours to complete.
Me: Okay, do you really think the average time to create all documents could be 2 hours?
SharePoint User: No, I don’t think the average could be that low, I think it is at least 6 hours.
Me: Good. So is your 90% confidence interval for the average time to recreate a document between 6 and 20 hours?
SharePoint User: That sounds about right.
Me: So based on this we could say that we have a 90% confidence interval that documents take between 6 and 20 hours to create.
If we are now able to measure the number of times that we recreate a document due to being unable to find it we can derive efficiency improvement.
Measuring the inefficiency
There are a number of challenges that this presents.
- How do you know you recreated the document if you couldn’t find the original?
- What things could be measured to give an indication of being unable to find a document
One that you may consider for this is using search in SharePoint, you may have noticed that when you search for a document it sometimes shows “View duplicates”.
These duplicates are not just based on being exact copies of the document, but based on the documents looking to the search engine as the same based on an internal document vector (mostly based on terms and numbers of terms).
The technical aspects of how this works aren’t of importance, what is important is that we have at our disposal a great technology that will allow us to find duplicate documents. This means that we have a way of identifying duplicate documents, and based on the document information the date they were created.
We therefore have a way to be able to report on the number of duplicate documents created by month over time – which using the accurate time needed to recreated the documents will provide us with one way of demonstrating an improvement in efficiency through the use of collaboration tools. It will also start to highlight where we are in fact storing actual duplicates, something we are also keen to improve on.
I plan to do a follow up post explaining how you would produce this report.
Other common measures
Other areas that using SharePoint and having good governance in place can help include
- Freedom of Information Requests – think about the ability to Find information
- Compliance – Security and audit capabilities can have a huge impact here
Sometimes the measures aren’t as specific as the example above; A customer told me that during a recent audit they had a dedicated team of people spend in excess of 3 months just locating and collating information for the auditor. And they were totally convinced, due to the sector they were in, that over the next 5 years they would have to respond to a similar request.
Therefore the legacy of what they are doing now it is possible to predict a significant efficiency saving by ensuring they align to the goals and vision.
In this post we have expanded on the outcomes phase of the 21Shift model and talked about how we might define some measures to support our desired outcomes. A way for us to feedback on what we are doing and if things are moving in the right way.
You may have read this and be thinking that this is too complicated! You have probably just deployed SharePoint, if you’re looking for complicated you have it right there!
These are simple techniques that help you think about the right things, to think about why you are deploying SharePoint and also helping you to clarify what it is you should be doing with SharePoint.
While we are in favour of measures and the feedback cycle to support continuous improvement this needs to be aligned to the first part of the 21shift model – the organisational effectiveness. There is a risk that people get trapped in the analytic stage, they continue to measure the simple things and fail to measure the intrinsic improvements, focusing on the numbers and forgetting about the system they work in.
Organisations need patience, they need to understand complexity and look at how to improve systemically towards becoming synergistic.
This post is part of the 21apps SharePoint Organisational Shift Model developed to be an antidote to these common issues in technology led projects.
An invaluable resource for anyone interested in understanding how to measure anything, something we often refer to in our own work.
Douglas W. Hubbard
We are proud to be part of the success that Paul Culmsee has had with this course; and our very own Ant Clay is the only other accredited trainer.