User Stories from the trenches

I couldn’t resist the topic. But for the record, credits to Henrik Kniberg for the “…from the trenches”-bit :-).

The past few weeks have been hectic, and I haven’t really had the time to post anything new in a while. The reason is that I’ve been in a phase here where we’re ramping up a project and I have been heavily involved in trying to form an initial backlog with an initial set of reasonably meaningful User Stories for that project. I.e.: we’ve been involved in getting a (very large) project “vision” down into a set of stories.

Our traditional method for this was to have the Product Manager (Customer) produce a written description of each feature that he wanted, and then he would hand that over to the Development Department where we (“we” being the team and the Scrum Product Owner: as you may know we use the somewhat controversial “Product Owner Proxy” construction in our organization) would take ownership of that written “spec” and break it down into User Stories that were prioritized and later estimated and delivered in priority order.

My problem with that approach was (and is) that the Product Manager usually (for whatever reason) spent alot of time writing these “functionality specs”.
I have felt that this goes against my core attitude/rule-of-thumb “Never document more than that you have to ask to understand” and that it is pretty "un-scrum". I’ve been wanting to find a better and more Agile way of moving from “The vision inside the head of the Product Manager” to “Something that the team understands, which can be prioritized and brought into the Sprint Planning meetings”.

In this case, with this development project being such a large one (a new product, with so many functions), it would simply have been an overwhelming task for the Product Manager to rely on our traditioinal approach and write all those "function specs" up front – at least considering the usual level of detail in those spec documents.

My idea was to skip the “detail specification” step alltogether and jump straight from the vision to User Stories, and thereby save time by having the Product Manager produce User Stories directly which later could be prioritized and further broken down. This would somehow force the Product Manager to skip details and thus not try to think of everything up front. Oh, and of course, we’d be forced to rely much more on face-to-face communication between the team and the Product Manager, which would be great: “two people by a whiteboard” is always a many times more efficient method of transferring information than by document, right?

So that was the plan, and this is what I have been struggling with for the past weeks (plus celebrating Christmas and New Year :-)).

I can’t say that it went great maybe, but I'm still pretty satisfied with the result so far. Anyway, I wanted to take this opportunity to really reflect over how things went and try to gather my own thoughts about it and nail down some lessons.

Here are some of the challenges with the past few weeks;

• I needed to find a way for the Product Manager to “empty his head” into something that could be used as basis for formulating some fundamental stories.
• I needed the Product Manager to understand what User Stories are and why they should be formulated in a certain way, what level of detail they should have, and so on.
• I needed to find a way – a method and tool – to document the User Stories which would fit nicely with the tools we use further down the road: we use the Backlog Manager tool that I have made available for download on this blog in a separate post.

I approached this by introducing MindJet MindManager to the Product Manager. I know there are open source and other free alternatives out there too, but I haven’t tried any of them myself.

I use MindManager for many different things, and I find it great for emptying my head down into symbols, boxes, notes and relations. So the idea was that it could be used by the Product Manager too, to get the rough set of functions and other thoughts and notes down into a visible map. That map could hopefully then be used to look at to formulate User Stories; e.g. by studying a subtree and all the nodes there to remember what needs to be covered by the backlog.

The map ended up looking something like this, but much bigger (the map below is just a very small dummy example, obviously):

In retrospect, we experienced a few major drawbacks with this method of using a mindmap to describe the vision of the project and have the Product Manager use it as basis for formulating stories;

1. Coping with keeping the map up-to-date with the stories: as stories are formulated, iterated, thought about, discussed and reformulated, after a while there are changes in the perception of how things are supposed to work. We failed at keeping the map up-to-date throughout this process. After a while the list of user stories conveyed a different picture than the map. However, I think it might be ok that the map is a “consumable” that can’t be trusted after a while; it’s just one tool used along the way and is not really an artifact by itself.
2. Ones stories were formulated, they ended up being too small. The initial backlog ended up containing much too many stories and each story gave just a too fragmented view of what should be delivered. The symptom of this was that it was very hard for the team to grasp the project as a whole, even with a lot of communication with the PM. The breakdown done was simply much too detailed and too early.

I think my main lesson here is to try to get the Product Manager to just summarize each desired feature as a Theme or Epic (see separate blog post), rather than in a whole bunch of small stories. If there are important details about the function that the Product Manager wants to mention he should write keywords as “notes“ to the story, and not try to write each as a separate story.

Also if there are things in the Epic/Theme which can be given different priorities, obviously those can be broken out as separate Themes/Epics. The ambition should however be to not break everything down in detail.
The finer breakdown, into “ready-ready” (see separate blog post) User Stories should be done by the team in cooperation with the Product Manager – not by the Product Manager himself.

So to summarize on the lessons learned so far:

• Get the Product Manager to use a mindmap to empty his head into, with regard to a certain vision of function(s).
• Get the Product Manager to look at the map and its subtrees and use that to write Themes (or Epics), formulated as User Stories (“As a I want to so that .”.
• Get the Product Manager to write those Themes/Epics in the same or similar tool that you will be working with in Development (e.g. if you can, directly into an empty Backlog Manager excel, if that’s the format you’re using).
• When the Product Manager feels he’s covered the most obvious and/or most important Themes/Epics, bring in the team. Let the team and the Product Manager together work on the finer breakdown – of the most important Themes/Epics.
• Talk alot about the Themes/Epics: between Product Manager and team. Let the Product Manager describe each Theme, if you have an updated mindmap you can use that as basis for a presentation about each, as part of the breakdown process.
  
• Break down the Themes/Epics one by one, and work face-to-face. Even if it takes the PM and the team a long time, I have a feeling that it is time saved in the end: because the team will get an excellent view of the backlog. A challenge here is to heavily moderate discussions about how to build the stuff: solutions shouldn’t be discussed at this point. That’s done when the stories enter the sprints.
• Don’t attempt to break every Theme/Epic down into smaller stories from day one. Make sure they’re explained and moved into the product backlog, but break them down in priority order: start with the most important ones. Leave the lower priority Themes/Epics for later sessions. Just because they’re big it doesn’t mean that you can’t estimate them. Remember: rather roughly right than precisely wrong.
• Come back to the bachy hklog after every sprint and review the Themes/Epics and let the team and the PM break them down once they surface in the backlog.

Here is an example of a subtree which could be used as basis for writing a Theme and later for discussing around as part of breaking it down into smaller stories;

Example of a Theme for the above subtree: "As a Administrator I want to be able to manage users so that I can control who has access to what."

Once the team and the Product Manager break it down together, they'd probably end up with something like the following user stories:

• As an Administrator I want to be able to add new users to the system so that each user can have his own personal account to login with.
  
• As an Administrator I want to be able to search for users and and be able to sort the resulting list in various ways so that it maximally easy to find what I'm looking for.
• As an Administrator I want to be able to Clone an existing user so that I create a new one with the identical data pre-filled so that I don't have to retype everything.
• ...and so on.

We’re still not done here and have some distance to go before we can say that the backlog is in an “OK” shape. I’ll revise this post and/or make additional posts with more thoughts and experiences as we go along.

Oh, by he way. I always appreciate feedback about your experiences. If you have any clever ideas or thoughts about this or how you work with it in your organization, please leave me some comments.

User Stories, Themes and Epics

In a coming (much longer) post about story formulation, I refer to the terms “Epic” and “Theme” and I just wanted to clarify what that means to me.
I figured I might as well make it a separate post since it’s somewhat a topic of its own.

• A “User Story” is a backlog item. nothing strange there, its everything you think it is. Its format is “As a I want to so that .”.
• A Theme (to me at least) is a large User Story.
• An Epic (to me at least) is a VERY large User Story.

For example, if you’re building “user administration features” your typical User Story, once it enters a sprint, might be something like “As an Administrator I want to be able to add users so that I can give a person an account to login with.”. That could be the “normal” size of a story that is far up in the product backlog, i.e. close to development. It’s pretty small.

Earlier in the process (further down in the backlog) it could be represented as “As an Administrator I want to be able to administer users of the system so that I can determine who has what access to the system.”. Notice the difference. This is a Theme. It’s much bigger and much less detailed than the story above. But it’s still not massive, and with some discussion it’s possible to figure out roughly what reasonably is hiding behind that formulation.

Earlier still (even further down in the backlog), the formulation could have been something like “As a System Owner I want various administration features so that I can control the behavior of the system.”. This story could be considered an Epic because it is so large and so generic and loosely formulated that it can cover a huge amount of different things.

Flexible scope - A magic bullet?

Everyone agrees that you can’t both have a rigid deadline and a rigid scope. We’ve been there, Done that, Got the g***amn t-shirt. Software Development is not mass production and last time I checked I wasn’t able to foretell the future. We want to avoid projects where both the scope and the date are fixed. That’s one of the first lessons every developer learns… And that is one of the key things that Scrum brings: you develop in iterations, you timebox and you ensure you have something to deliver all the time, especially release day. Yay!

Timeboxing means the date should be regarded as fixed. Wow, there’s an easy sell to upper management…! But as a consequence we have to allow the scope to flex… Simple enough? Well, allowing the scope to flex might not be as easy as it seems. “Flexible scope” as a concept is easy enough to understand, but it’s not something that happens by itself just because you say “we do Scrum”, develop in iterations with beautiful sprint planning sessions and sprint reviews.

I’ll illustrate with a scenario that I’ve seen in reality several times:

Let’s say you’ve started using Scrum, and one of the key selling points to upper management was: “In Scrum we never move the deadline, we promise to try to always have something releasable at the deadline. I can’t promise exactly what, but we won’t ever need to move the date”. Good start!

So, now you start working. The team is working on a release of (let’s say) two major features (each consisting of several smaller “parts”). The Scrum Product Owner and others work on breaking down the scope(s) into User Stories. But they’re not really familiar with Scrum and the stories they produce are poorly formulated, and are done without them really understanding what a User Story is and what to think of.

Still, eventually a product backlog is ready. But, again since the Scrum Product Owner might not be that familiar with Scrum there is not much effort or thought put into the prioritizations of the stories in the backlog. Anyway, the team starts their sprint with a sprint planning meeting and grabs whatever is on top of the backlog.

Once rolling, the team execute the sprint with little or no regard to the priorities. Everyone grabs a story each and start working.

Work continues along those same lines sprint after sprint. And stuff actually do get done. Progress is OK, people feel OK. So far so good?

Now, when you approach the intended release date, you look closer into what’s been done so far and you conclude that there are still several critical things that aren’t ready yet, or that maybe haven’t even been started yet. In addition, you conclude that there are critical tasks remaining for each of those stories which you previously thought were “Done” – tasks such as integration testing, string translation, code reviewing or similar. Ouch.

At this point, what are your options?

1. The "easy" way out is to postpone the release date, i.e. effectively breaking the timebox rule. …Scrum breaking down. Oh misery…! And you had such an easy sell to upper management with the fixed-date thing... And all you do is move the date and then have the same problems once more, but that time you've sortof used up your goodwill and no longer have the luxury (?) of being able to postpone the date...
2. Crunch! There's that word again... But you think "Just this time… This release was special...". You make up excuses about why you’re where you are, and call for overtime. Have people come in a little earlier and stay a little later. Have them come in those last few weekends. Not to mention those last days before deadline. So, the team works overtime, they put in extra hours, and not surprisingly quality suffers, people suffer, shortcuts are taken everywhere… Anyone recognize this?
3. Flex the scope by lifting out the entire feature from the release. Since there are still critical stuff remaining you don’t have the option to just abandon parts. All you can do is lift out the whole feature. That doesn’t give such a good impression of the whole “flexible scope” mechanism, now does it? “Yeah well, in Scrum we don’t move the date, so we have to let the scope change, and that means removing a feature from the release. We have to allow that. That’s Scrum.”. A job well done. No, this is not what “flexible scope” means.

Those are really the only three options. And what’s even worse is that this is probably not discovered until very late. With only days or a few weeks (best case?) remaining until the scheduled release date… This is Scrum failing big-time!

So, the point I want to make here is that we need to be very careful so that we really maximize scope flexibility. The main mechanisms in Scrum which are affecting this are:

1. Strict prioritization
2. Ability of adhering to Definition of Done == Releasable.
3. Skillfulness of breaking down a scope into user stories

To me “strict” means that we both prioritize AND stick to it, by actually working on things in priority order. If we don’t have a prioritization we can’t really say that we have done all in our power to have everything “critical” or “most important” (or whatever value you want to put in the prio) ready by the time the timebox ends. If you’re not careful, the “completed” part of the scope may be any combination of important and less important things, and (what’s worse) the same is true for the stuff that isn’t done yet.

Prioritizing the backlog needs to be intelligently done, and only 1 person should be allowed to have the final decision about a story’s priority: the Scrum Product Owner. I recommend thinking of the importance in terms of “Business Value”. But be careful; it doesn’t only mean value as in dollars and cents (or Swedish Krona and Öre ;-)). The keyword here is Business. Think of it in terms as Value to your Business, including your own internal business of developing stuff. So things that for example are “architecturally significant” (I’ll post an article separately about this topic, some day) might get a high Business Value because it is important for the architecture and the consecutive development of other stories - even though the story itself may have less of an obvious value to the end user, the customer, your accounting department or your shareholders (but remember: if it is architecturally significant it is most likely valueable to the customer and end user too, it's probably just a matter of thinking about it hard enough and being able of formulating it in words).

The Definition of Done applies to how you “complete” stories. I’ve posted entries here in the past about this and I won’t go into details here. I just want to point out that adhering to the Definition of Done affects your ability of flexing the scope, because you ensure that you stay in control over what’s remaining; you stay on top. The things that have been done are really “done” and are really releasable, there are no hidden piles of work under the rug. I hate surprises and Definition of Done is my best friend :-).

A largely contributing factor both to your ability of adhering to the Definition of Done, and to your ability of following a prioritization, is how good you are at breaking the scope down into User Stories. Don’t regard the scope as a big monolithic lump of functions (e.g. the waterfallish term “Requirements”). Instead, you need the stories to be independent, finishable, valuable and estimatable. A good guide on the way of thinking you want to have is the I.N.V.E.S.T. acronym (stories should be Independent, Negotiable, Valuable, Estimatable, Small and Testable). Writing stories is a skill that you have to work actively with and continuously try and improve.
Don’t be alarmed though; it’s not rocket science. Some breakdown is better than none, and you shouldn’t make such a big deal out of it. Learn by doing. Try different approaches to formulating stories, try the INVEST acronym, try other ways. As long as you understand that the “quality of the breakdown” will affect a lot.

So, to wrap things up: with intelligent scope breakdowns, by following the prioritization and the Definition of Done, we can be more than reasonably sure that the e.g. 85% of the scope which is “ready” when the timebox ends, is really ready, and the remaining 15% is not half-started and we don’t have dependencies between the ready 85% and the remaining 15%. We know we can release something – and that something is actually something valuable (hopefully even the maximum value).

There. Long post this time. But I like this subject. And it's an important one. I’d be glad to learn about other peoples’ experiences. Cheers! My food is getting cold.... ;-)

Slices (Verticals), User Stories and Scrum

We try to divvy our stories in a way that they span from top to bottom; through all layers of the architecture. We call such an isolated part of a function or project, a “Slice”. It is also commonly known as “Vertical”.

But why develop in slices, or verticals, why even care about it, why not just do the User Story regardless and forget about if it is a horizontal or vertical or whatever…? Well, we’ve tried various approaches here. The one thing I am certain of is that this is hard. It requires skill to be able to produce “good” user stories. And not only is it hard, it is also very important: because badly formed User Stories can wreck velocity. Properly shaped and formulated User Stories is one of the basics, a requirement, for a well-functioning sprint execution. So, Hard + Important means that we have to focus on learning it and really spend time thinking about it – and continuously evaluate our approach and view of the matter.

Ok, so then why vertical and not horizontal?
If you do things horizontally, isolated to each architectural layer, then it will be hard for you to actually deliver working, testable code. By doing Slices you are forced to deal with every layer right from day 1. This foundation will then be built further upon, as you iterate the software with every new slice. Also, working vertically – meaning involving every layer in every slice – will often require different competencies (for example UI/web, C++ and SQL/db). Suits nicely with our cross-functional Scrum teams, doesn’t it?

To use Lean terms, it’s about decreasing the “batch size” and ultimately minimizing cycle time; i.e. the time it takes until you deliver something valuable to the customer. If you work horizontally it will take longer until the customer has something that brings value. A complete user interface, but empty of all functionality, is rarely valuable to the customer (and rarely “complete” for that matter, until you’ve built the underlying stuff). And it is definitely not something “releasable”. Remember that you want to strive towards having something releasable every sprint. Sometimes (often?) it doesn’t make sense to release “half a feature” (e.g. “Add User” without “Edit User” functions), but at least you need provide that option and let the customer decide what he wants and doesn’t want: don’t make the decision for him by implementing horizontally.

So what forces are in play here, with regard to stories, architecture and Scrum? Well, let’s recap;

• We’ve just concluded that we have a requirement to build vertical increments of the system: Slices.
• We also have a requirement to “finish” (Done-Done) things, with each sprint.
• We even have a requirement to be able to “Release” at the end of every sprint.
• And we have a requirement to think iteratively, i.e. we should come back continuously and refactor our code (continuous refactoring – see blog post below)

Don't document more than that you have to ask to understand

Jeff Sutherland writes in his most recent blog entry about Agile Specifications. Interesting stuff!

I have rule of thumb that I use and try to teach everyone, when it comes to documentation: Don't document more than that you have to ask to understand. It is, I think, something to remember when it comes to writing agile specs, user stories, etc. Spending lots of time writing fancy specs and try to catch every little detail is just a waste of time, because I think it causes a false sense of trust in the spec. You won't succeed just because you discribe every little aspect of some feature. What will let you succeed is if the developers manage to understand it well enough to build it. And communication is key here. So don't write fancy specs just for the sake of it. Communicate instead! Ask! But as Jeff describes in his article above, of course you should document enough. We do write specs too (I'll describe our "scoping process" some day). But the above rule of thumb should always be kept in mind...

Picking stories at sprint planning = Drawing The Line

My thoughts today is about how the team picks stories at the sprint planning meeting.

One point of the sprint planning is for the team to be allowed to pick the amount of work they feel they can cope with, and commit to, for the coming sprint. As a preparation for the sprint planning meeting, the product owner goes over the backlog and revises the story prioritizations based on most recent feedback/input from sprint reviews etc (whatever affects the priorities).

Our approach up until recently was to let the product owner present and make available (e.g. on a whiteboard) the X topmost stories in the backlog. Those stories would be considered "up for grabs"on the sprint planning, but without relative priority. The team could freely pick any of them, in any order. It sounded so good on paper; the team has complete freedom over which stories they pick, yet the product owner has a say about it by means of him prioritizing the backlog prior to the sprint planning meeting.

The pitfall with this is that since the team picks any stories from the top of the product backlog, the stories enter the sprint backlog without priority; or at least the priority has less of a meaning now. And it is therefore tempting to just regard the set of stories in the sprint backlog as one big unordered set of stories - all which must be completed. And as a consequence you lose one of the most valuable concepts of Scrum & Agile: focus on the most important thing first. Plus, it leads to people in the team isolating themselves with one story each, which means the cooperation effect is lost.

Don't do what we did!

So how should you do it? Well, the product owner should prioritize the stories in the backlog, and they should be ready-ready (see previous blog post). On the sprint planning meeting these stories are put in a list and the team can not just pick any of them. Instead, what the team does is to draw a line somewhere. That's it. This drawing of The Line represents their committment. Everything on one side of the line is what they commit to delivering, and the rest is left for consequtive sprints; or whatever the product owner choose to do with them. In our case, we should have the stories printed out on index card papers, and put on a whiteboard from left to right in descending priority order. The team then draws a line using a whiteboard marker. Simple enough! :-)

Ready-ready: the Definition of Ready for User Stories going into sprint planning

You all know what “Definition of Done” means. Just for the hell of it I’ll recap it anyway; What it refers to is a state of a chunk of work. The state is also referred to as “Done-Done”. The purpose of the Done-Done mechanism in Scrum is to ensure that those things completed in the sprints really are completed in a commonly acceptable, controlled, premeditated and agreed sense. For example, nothing should be considered “completed” until it is e.g. tested, reviewed, committed and merged – e.g. “Releasable”. Without the definition of done we risk building up a mountain of stuff that we push ahead of us, which eventually (e.g. release-day) will kill us – or at least give us a hell of a fight. What’s even worse, without the definition of done we not only build up such a mountain of stuff, we also have no way of knowing how big that mountain is until we start getting our hands dirty and digging in. And what’s even worse is that many of the things inside that mountain will take so much longer to do later than if performed immediately day 0, because the longer it waits the harder it becomes to fix, to understand, to find, to remember, etc.

So, I think we’re all in agreement that we need a definition of done for the work we ship out of a sprint. Now I think we need a similar mechanism for the work taken in to the sprint. A mechanism which ensures that the stories taken in is really ready to be taken in. Let’s call this state “Ready-Ready”.

First of all, what does “taking in” mean? Well, it means “available and up for discussion on the sprint planning meeting”. So the definition of ready needs to describe a state of stories which they must be in, in order for them to be eligible for discussion on the sprint planning. Before they have reached the Ready-Ready state the stories do not exist and will not be considered for sprint planning. I.e. the top part of the product backlog has to contain only stories which are ready-ready.

Why? To save time of course: to spare the team tedious discussions about stuff that isn’t sufficiently thought-through. To let the team focus their valuable time and energy on things that are as “safe” as possible.

So, the definition of Ready should be;

• A user story exists which points out the actor, describes the targeted feature and the purpose of it. The story should be formulated like this; As a I want to because . The first two are often easy to pinpoint but the purpose is often overlooked, because it feels obvious. But is it? The idea with writing down the purpose is to give the reader a good feel for the context of the feature – what problem it solves. Not only what it is but also why. This way, the reader can form his own opinion on how to implement it. It gives the reader a deeper understanding for the story.
• The formulation of the user story is general enough for the team to have the flexibility of delivering it in increments. Don’t describe every detail. Describe roughly what you want but leave the rest for discussion.
• The story is small enough to fit inside a sprint. Stories larger than that need to be broken down. Breaking down and reformulating stories have to be done before the sprint planning. Obviously, a story cannot be completed in a sprint if it is larger than a sprint. So care should be taken to ensure they are small enough.
• Each story has its own unique priority in relation to every other story in the product backlog. This is to ensure that the team can work on the most important things first, and to force the product owner to make intelligent decisions about the order of stories.
• The story has a description of how it can be tested (demoed). The description need to give the reader a good sense of what determines if the story is completed or not. It should be e.g. the acceptance test that is described here. It can be used as a description of how to demo the story too.
• The story has been estimated (in story points). This should preferably be done by the whole or parts of the team, but at the very least it should be done by individuals with the same understanding of the system and the domain as the team who will eventually be working on it.

So in our case we should have meetings prior to the sprint planning meeting, where the product owner meets the teams and sits down and discuss the product backlog. Don’t forget to time-box the meeting. The goal of the meeting is to deem stories as ready-ready. It will require that some stories are broken down, reformulated, etc – and estimations are done using poker planning. A tester or test leader is required to attend here too, so that the formulation of the “how to demo” is really testable and meaningful.