NetBeans Quality Metrics Report
I have found an interesting example of using metrics in open source by NetBeans community. NetBeans is a free, open-source IDE for software developers. NetBeans developers are putting up all bug metrics publicly on a special QA Metrics Report that is updated daily.
On the report, you can see:
- chart on weekly issues reported vs issues resolved.
- issues reported by Sun (I guess it should be called Oracle now) vs Community
For me, there are some aspects missing to have make this report useful. First of all, I would like to see statistics per major releases of NetBeans. It is interesting to understand how major releases compare to each other. Metrics like number of escaped defects, defect detection percentage and defect resolution efficiency would be great to know. Second, I am missing the number of currently unresolved defects broken up by priorities.
Interesting if non open source commercial products would also follow this type of opennes. Would be cool to know how many bugs are there in product like: Gmail, Facebook, Salesforce…
Reading “Seven Pragmatic Practices To Improve Software Quality”
Over the weekend I have spent some time reading the paper “Seven Pragmatic Practices to Improve Software Quality” by Mike Gualtieri. Executive summary:
All software has bugs. Application development teams do the best they can to avoid and fix them. But developing quality software is hard. Why? Virtually all application development teams are squeezed by the need to develop and change applications faster, but few have the time to implement a full-blown, guru-based quality program. What they need are shortcuts — pragmatic changes that application development teams can make to improve software quality without getting the brass and rank and file in a tizzy. Forrester has identified seven pragmatic changes that can have a big impact on improving the quality of the software you deliver to the business: 1) define quality to match your needs; 2) broadcast simple quality metrics; 3) fine-tune team and individual goals to include quality; 4) get the requirements right; 5) test smarter to test less; 6) design applications to lessen bug risk; and 7) optimize the use of testing tools.
What I like the most about this paper, is that it start with definition of Software Quality. It is so often, that people who try to improve quality, forget to take time to understand and communicate what do they mean by quality. Here is the definition by Forrester:
Software that meets business requirements, provides a satisfying user experience, and has fewer defects.
I think this definition is really great. It pleases stakeholders (“meet business requirements”), users (“satisfying user experience”), and adds internal quality view (“has fewer defects”). Perfect.
Talking about 7 proposed practices, I think these are too generic. I hoped to to read more specific suggestions or even a checklist. E.g. Testing Smarter or Get Requirements Right sounds great, but doesn’t answer how. Don’t get me wrong, I think the list is correct. It just needs more analyses from the team side to be applied. As you may guess my favorite practice from the list is “Broadcast Simple Quality Metrics”. I would only add that you should automate collection of those metrics.
Definitely recommend to add this paper to your reading list.
why tracking of advertising metrics is more popular than software development metrics
There was a short discussion yesterday in our office about why advertising metrics are so much more popular compared to software development metrics.
It is indeed interesting how advertisers and marketers pay so much more attention to metrics. When you read marketing blog, you can feel that in this domain specialists are obsessed about metrics and analytics. Metrics like: cost per lead, conversion rate, customer aquisition cost are well discussed and understood by the community.
And what about metrics in software development field? Software metric term definition is missunderstood, there is no industry strandard or well accepted metrics, application life-cycle tools include only very basic statistics that is far away from analytics that can help development teams.
I bet, that at least 50% of those doing online advertisement are tracking metrics, and probably not more than 5% of software team track software metrics ( I don’t have stats on this, so probably I am totally wrong on this).
There should be a reason for that.
First, that comes to my mind, is the fact that advertisement is more aligned with money. You spend $100, and you expect to earn $200 back. And you need metrics to see if you actually achieve this goal. Typically in marketing it is fast to see the results, so you can feel effect of your marketing campaign already in few days or weeks after launch. With software development, though, especially with enterprise software, it can take months or sometimes even years to feel the outcome. Having the feedback loop so long, can make collection of software development metrics not so cool.
But, im my opinion, actual reason is that tracking advertising and marketing metrics is much easier. Google has made it a standard to embed stats and analytics into the way you do online advertisement campaigns. There are bunch of tools like Clickable, Covario, Marketo that make collection and tracking metrics easy and enjoyable to do. For software development project managers, though, tracking metrics is often associated with the pain of weekly or monthly manual data collection, negative attidude of developers towards metrics, and lack of public information available on metrics.
I know, it is impossible to solve this problem with just one blog post. But, I truly hope this situation will change over the time with introduction of new tools on the market and with better awareness of metrics benefits in the field of software engineering.
What do you think?
Metric: Time Between Escaped Defects
Very interesting metrics, though, often not tracked is Time Between Escaped Defects. This metric shows you how often new defects are identified in released versions of the product.
To calculate that metric, you should be tracking following fields in your bug tracker:
- Defect registration date and time
- Ideally, defects are registered to different component so you can compare the value of metric for different components
calculation of the metric is not very simple, if you want to do it manually. Though, simple excel can help. 1: You filter out all escaped defects from your bug tracker. If you are interested in this metrics for specific period then filter out only escaped defects registered during this period.; 2: Now sort all defects by registration time. 3: If you are using excel, then create a new formula column that would calculate time between defect registration time and previous defect registration time. 4 Now when you have a table of values for Time Between Defects and you can do basic statistics like: calculate mean, min, max, ..
How can this metric help you? This metric is useful to estimate maintenance effort for the product. If you know, that there are 10 defects found on released versions of your product every day, you probably need a dedicated developer or two to maintain your product. If it is just once a week, then probably product team can handle the bugfix. Also, tracking the trend that this number is always decreasing can mean your product quality is improving.
For visualisation, I suggest 2 charts:
- bar chart, with Mean Time Between Escaped Defects for different components or products that your are working on. If you don’t have multiple products, then try to gather statistics from other companies or teams that develop similar complexity products.
- trend line chart to make sure you are increasing the gap between new defects are found.
Using line chart to visualize metrics
I find choosing an appropriate visualization for metric as one of the most difficult tasks when designing a report or dashboard for software organization. There are often numerous ways to present the same metric. Also different interests of dashboard users make the correct choice even more difficult.
Line chart is always the first and most simple choice. It is best to represent the trends of the metrics over time. Line chart It is very simple to implement in any spreadsheet or any other tool you are using. One of the best implementations of line chart is done by Google Visualization API - http://code.google.com/apis/visualization/documentation/gallery/annotatedtimeline.html .
Though, I like the line chart and we use it in our software a lot, I believe it is often overused in dashboards. Take a look at the example below taken from Google Analytics default dashboard:
It supposed to visualize the metrics Number of Visitors to my blog. Even though, metric is very interesting for any blogger, but what this charts tells me? Not much. I see there was a peak of visitors, due to the popular blob post few days ago. But do I need to see that every day? When I open my web analytics dashboard, I am interested to understand the current status and not just see the trends. That’s why I find this visualization much better
It brings me the most important information highlighted. And it gives me an idea of different aspects.
What is your favorite visualisation for metrics? Let us know…
Real-time accuracy in software metrics
With all the hype around real-time data in web and other domains, lots of time Programeter team gets requests and questions about importance of real-time accuracy when dealing with SDLC metrics. So I wanted to share some answers and ideas on applicability of real-time accuracy in software metrics domain.
According to Wikipedia “Real-time data denotes information that is delivered immediately after collection. There is no delay in the timeliness of the information provided. Real-time data is often used for navigation or tracking”. Translating that into software development domain, it can sound like: Real-time metrics that help you to timely react on events happening in your software project and adjust your plans”.
So does software project tracking needs also real-time data, or in this domain it’s not so important? First reaction, is that it doesn’t matter if you get metric value in a second after measurement or if you get it in an hour or day. I hear this answer often from managers of projects with long iterations. The delay of few days doesn’t influence them a lot.
Same time, while being a product owner myslef, my need for fresh metrics is very high. We have very short iterations. And if team spends one day of doing wrong things, because we were not notified of something real-time, it can influence the outcome of iteration a lot.
But don’t just rush together with the hype and start requesting your metrics team to deliver all metrics real-time. It is important to understand that it comes with the cost. To have real-time metrics you have to:
- calculation of all the metrics should be fully automated;
- Metrics should be presented using online dashboards, instead of the weekly or monthly reports delivered to stakeholders by email;
- Proper infrastructure that support collection of those metrics in a way that as soon as metrics change happens it gets delivered to the dashboard.
But to be honest, I still don’t have solid opinion whether real-time metrics are important to most of the software teams. So, please, share your experience with implementing metrics in your company in comments.
Reading “12 Steps to Useful Software Metrics”
Yesterday I have a luck to find a great old article by Linda Westfall. It is date 2005, but it is on of the most useful reading for anybody trying to implement a metrics initiative in their team or company.
The full paper is included below. But if you are lazy, just browse through the 12 steps to get an idea:
- Identify Metrics Customers
- Target Goals
- Ask Questions
- Select Metrics
- Standardize Definitions
- Choose a Measurement Function
- Establish a Measurement Method
- Define Decision Criteria
- Define Reporting Mechanism
- Determine Additional Qualifiers
- Collect Data
- The People Side of the Metrics Equation
With tools like Programeter you could automate or simplify some of the steps. But, don’t try to skip any of those if you wish to have success with software development life cycle metrics.
Average Age of Open Defects
Having covered yesterday the Average Time to Fix metric, I thought it would make sense to also introduce you to Average Age of Open Defects metrics. Those two, in my opinion, should be tracked together to give you the full picture.
To calculate that metric, you should be tracking following fields in your bug tracker:
- Defect registration date and time
- Defect resolution status
then calculation is pretty simple. 1: You filter out all unresolved defects; 2: You calculate time between registration and now; 3: Sum all the calculated times and devide it by total number of unresolved defects.
Good visualization could be a set of 2 charts: 1: bar chart with current values for each priority; 2: line chart with historical trend of the metric. For Average Age of Open Defects I believe key is to track the trend of the values. In most cases it makes sense to constantly reduce average age, which indicates that you are fixing the defects.
SDLC Metrics: Average Time to fix
Today, I wanted to get back to good tradition of this blog and bring up another definition of software development life cycle metric. This time it is a Mean Time to Fix. Mean Time to Fix (in other engineering fields often called Mean Time to Repair), as the name suggest shows you the average time that it takes you to fix the registered defect.
To calculate that metric, you should be tracking following fields in your bug tracker:
- Defect registration date and time
- Defect resolution date and time
then calculation is pretty simple. 1: You filter out all resolved (or closed) defects; 2: You calculate time between registration and resolution; 3: Sum all the calculated times and devide it by total number of resolved defects. In most cases you are interested in this metric for given period of time. In this case, on step 1 you should filter only defects resolved over the period you are tracking.
Whether you have an SLA for bugfixing or not, higher priority defects are fixed faster than low prioirity ones. So when tracking this metric be sure to collect number for different priorities. Sure P1 and P2 should have as fast resolution as possible, but it is also important to see that lower priority issues are not forgotten.
In addition to priority it wise to differentiate fix time for defects found in QA vs escaped defects. Time to fix defects that customers face is more critical in terms of product quality.
What’s the use of the metrics? Depending on your product, it shows you have your customers feel about your quality. For some customers, speed of resolving their issues is key criteria of quality. Also high value of time to fix can give an idea of how difficult / how costly it is to fix issues in your product. If trend is growing higher, than probably you have an issue with your product architecture or code quality. Another use case, is too keep an eye on lower priority defects so they get resolved in a reasonable time.
Why developers are afraid of software metrics
Recently, we have received a comment from Alexander Beletsky in which he asks if we have experienced protective behavior from developers and managers while pitching our products. Alexander explains how he feels that both Programeter and devPulse can be used to watch over developers.
While the straight answer to the Alexander’s question is “yes, we have experienced protective behavior while pitching Programeter and for devPulse we haven’t heard such concerns as this is a communication tool for developers”, I thought that this topic is interesting enough to deserve a separate post.
So, first of all, let’s consider why would a developer be afraid of software metrics. First thing that comes to my mind is the fear that my personality will be reduced to a single stupid number and I will be harassed by the management based on this number. To address such issues we always prepare a metrics workshop for our customers and main tagline is that “software metrics give you chance to ask questions, but they don’t give you answers”. Some other guidelines are that metrics implementation program should be as open as possible: all roles should be involved in it; goals, reasons and results should be openly discussed. In my opinion, all these are common sense. Decent manager would know/understand that even without special preparation. But, somehow we, developers, tend to dramatize things - our managers should necessarily be pointy-hair bosses with all kinds of freaky ideas :). Let’s agree here on one thing: if you have such manager then you are already screwed with or without metrics.
If your boss is adequate and you are really doing your job, then you have nothing to fear - metrics will only help your boss to understand you better. Moreover, metric will help you to know yourself better and for example experiment which working techniques work for you and which not (e.g. pomodoro). Finally, metrics will help your team to become more productive and efficient. It works similarly to performance optimization of an app - it is impossible to troubleshoot performance without knowing where are bottlenecks and why they exist - you have to start with a measurement first (read metrics) :).
And of course, Programeter is not about one single metric - au contraire - we provide set of metrics that describe the whole software development lifecycle from different perspectives: defect removal efficiency, defect detection percentage, code churn, know-how, iteration burndown, product size, test automation rate and others.
I hope that with this post I was able to lessen your fears and skeptisims about metrics. If your particular idea was not addressed - let me know in the comments - it will be fun to have a discussion over it! :)
