Estimating & prioritizing the product backlog

One of the things that I’ve been struggling with getting my head around since I started using Scrum is; how much effort do I put into estimations of the product backlog, and do I or don’t I estimate all at once? And do I re-estimate stories as I go along and learn more?

Originally I’ve been under the impression that I shouldn’t attempt to estimate it all up front. It's easy to argue that it is rather un-agile to do so: because I would be spending effort on low priority stuff, which is a big risk for waste. On the other hand, if I don’t estimate the backlog up front, I can’t tell its entire size and consequently I can’t (potentially) tell what I will have ready by when.

So far, I’ve been trying a couple of different approaches. One of my favorites has been to estimate in priority order up to a few months worth of stories (a "handful" of sprints). The remaining unestimated stories I apply an average size to based on the estimated ones. I know, it’s risky, but it’s at least something. And obviously, as time passes, I estimate more and more; trying to keep about 2-3 months worth of stories estimated at all times. For some time this has been my approach to release planning without estimating everything up front.

A problem I only recently realized I’ve had is that this approach requires me to prioritize before estimating; and consequently my prioritizations can’t take size into consideration. My prioritization has only been made based on “value” (whatever that is) without regard to cost. Is that optimal? I think the answer is No. Given I have a choice prioritization should take size into account. Or at least, if I don’t take it into account it needs to be a conscious decision not to. Which brings me back to my original question; how do I do it, do I estimate everything up front or not?

Recently, I’ve decided to try changing my approach (inspect and adapt, right?) and actually estimate the entire backlog up front, before prioritization. The good thing is that estimating in story points is very quick (the team usually doesn’t have to spend more than a few minutes per item) so the time spent on (potentially) low-priority items is minimal.

Once I have the story point estimates of all backlog items I use a prioritization technique called “Theme Scoring”. Let me try to describe it with a simple example;

The first step is to decide on a couple of aspects ("selection criteria") to judge stories on; for example “Improves user interface”, “Simplifies maintenance”, “Brings revenue in Q3” and so on. The combined criteria will represent what you call “business value” so whatever you put in that term should in one way or another be reflected by those criteria that you pick. Try, however, to keep it rather simple e.g. up to a maximum of 4-6 criteria or otherwise your scoring will become very tedious and the effort you spend on scoring (and consequently the risk of waste) I think will be too great compared to the benefit.

Once you’ve decided on your criteria you need to weigh them relative to each other. Maybe “Brings revenue in Q3” is the most important criteria to you right now, so let’s give it the weight of (for example) 5. “Simplifies maintenance” is not that important right now so you give it the weight 2 (it’s less than half as important as the revenue criteria). “Improves user interface” is almost as important as the revenue aspect, so you assign it the weight 4. Here are your criteria and their weights so far; “Brings revenue in Q3” = 5, “Improves user interface” = 4 and “Simplifies maintenance” = 2.

The next step is to go through all of your backlog items, one by one, and score them on a scale from 1-5 in terms of how they affect each of your selected criteria. For each criteria you can usually identify a “baseline story”; a story that gives you a fair amount of value in terms of that criteria, yielding in a 3 point score on that 1-5 scale. There should be stories that yield in a higher score as well as lower. Stories should be compared to that baseline story for each criteria to determine if it should get a higher or a lower score than the baseline.

This could be your result after going over all criterias for all stories;


The numbers marked red are the baseline stories for each criteria. The black numbers in the criteria columns are grades I assign each story on a scale from 1-5 relative to the baseline. 1 and 2 means much lower and lower than the baseline, and 4 and 5 means higher and much higher than the baseline. A grade of 0 means that the criteria doesn’t apply to the story. The Score is simply your grade multiplied by the weight, for each criteria.

Given your selected criteria and their weights, in the example here Story 3 is the one giving you the most value (37), followed by Story 2 (18) and then Story 1 (17), Story 5 (8) and Story 4 (6).

Now, it is time to compare value to cost (estimated size in Story Points);


So far this technique has been possible for me to do all along, even while I have been lacking estimates of parts of the backlog. But the following part – taking cost into the equation – obviously requires me to have the estimates. Putting the Benefit in perspective to Cost is a great idea; it gives me the opportunity to maximize benefit while minimizing cost.

So, divide Value (your total score for each story) with Cost (measured in story points) and sort your backlog by the result: the higher the value the more value I get back on the effort put in:


The above table is the resulting suggested prioritization based on the Theme Scoring technique and on calculating value/cost. Obviously it is worth noting that this is a suggestion only. It is an aid that I can use when prioritizing my backlog, it’s not something that replaces my thought-process – I still need to think and consider every item and e.g. be careful that I don’t miss any dependencies between stories (yes I know, we shouldn’t have any dependencies between stories, but sometimes we do anyway! :-)).

As always, I’m interested in input on how other people do things…!