Estimation - why and why not?

I work for a project which follows agile methodology for software development.

My team has Programmers, Business Analyst, Testers, Automation engineers, Project Managers, Subject Matter Experts and Customer.

When we need to develop a feature, we need to estimate and find out when that feature would be complete.

When we need to plan a feature, we ask estimates only from programmers. We don’t really ask estimates from other teams like business analyst team and testing team.

Are we doing a bad planning by not asking estimates from analysts and testers?

The estimation is one of the attributes requires for planning. We do estimate so that we can plan. So, when we ask estimates from analysis, testing or programming team, it should help us to do planning. If estimation does not help us in planning then we should not estimate.

We follow agile methodology, which is more or less similar to Just In Time (JIT) method of manufacturing. We like to keep our inventory level as low as possible. For us, inventory is a piece of requirement, which is termed as “story”. We need to ensure that we have minimal number of stories in pipeline. My pipeline starts when a business analyst starts breaking a “customer need” or "feature" into stories. The pipeline ends when story development is complete and it is ready to go live.

I need to plan so that my pipeline is thinner. I should pick up the inventory that i can finish. Thus, I need to know how much programming team can consume in a given time period. Thus we need programming estimates. I also need to know numbers of BAs and Testers needed to support the work that programming team can take. Hence, BA/Testing estimates.

Now, let’s take a different point of view. JIT didn't become as popular in USA as it was in Japan. JIT demanded a strict discipline during planning and implementation phase, which many companies in west were not able to adapt. Though, western countries could successfully able to adapt an ‘Avatar’ of JIT called “Theory of Constraints (TOC)”. The basic principal behind TOC was similar to JIT -keep minimum inventory in pipeline. But, it was rather easy to adapt compared to JIT.

TOC suggested that the team will move as faster as speed of your bottleneck resource. In a highway, if overtaking is not allowed, the traffic will move with speed of slowest truck and not with speed of a sports vehicle. Thus, you need to ‘estimate’ the throughput only for the bottleneck resource to find how fast the team can go.

Applying the same analogy to an agile software development project, we can chart out project plan based on speed of bottleneck resource. To do that, first, we need to find the bottleneck resource. Though, we don’t need estimates from any non-bottleneck resources, we like to closely plan capacity for other resources so that those resources don’t change in a bottleneck. If a non-bottleneck resource changes to a bottleneck, then throughput will be based on speed of new bottleneck. The old bottleneck may not remain that relevant in this scenario.

How do we know that a particular resource is becoming bottleneck? The stories will always queued up in-front of bottleneck resource. If your testing team is a bottleneck, you will see lots of stories lined up for testing.

If you start with an assumption that your programming team is going to be the bottleneck, then you should do only programming estimates. If you take programming estimates for planning, ensure that small amount of inventory (stories) always piled up before programming team. This is to ensure that bottleneck (or critical resource) is never having dearth of work.

The BAs should pile up stories for programming team. The pile in-front of programmers should never be empty. Thus, it is better to have some extra BA capacity so that BA team should always be able to satisfy the need for Dev team. The BA capacity should be increased or reduced based on pile of stories for developers.

The Testers should always be free (yes, free) to pick up stories for testing as soon as a programmer completes the stories. There should not be any pile in front of testing team. Again, the testing team capacity should also be closely monitored and adjusted as required.

We will agree that the BA and testing team capacity cannot be adjusted on daily basis to achieve the same. The resources take time before they can join the team and be productive. Thus, for a new team, the BAs and testers should be added in a ratio to the programmers and then, should be adjusted to suit the need for the team for first few months.

After these adjustments to BA and testing teams, the BA and Testing estimates does not really required for the planning. The programming team sets the pace and BA/testing teams always able to meet that pace as they are extra in capacity.

Based on above analogy, I don’t see any reason why any team needs to take estimates from BAs and testers unless they are real bottleneck in the project!! If we think that BAs (or testers) are bottleneck (i.e. critical resources) then we should take estimates only from BAs (or testers) and stop getting estimates from Programming team, as BAs (or testers) will decide when project (or requirement) would be complete !!

How do I decide optimum release size?

So, my project sponsor wants to do a production release every month. He feels that we are not agile enough as we are doing releases only once in 3 months. Our team size is 100+.

His core reason is : He needs to give the functionality to his customer with shortest lead time possible. Also, he likes to keep giving functionality to his customer one by one rather than in one go.

If we make release duration too short, it will not allow us to complete some features scheduled in the release. This is an extra overhead for the release when a feature is not production ready, as dev team needs to switch off that functionality and QAs needs to ensure that all workflows works fine after switching off the half baked feature.

So, what are the factors that i should consider before deciding on a release size?