Home Industry Data Conferences Links Article 1 Article 2 FAQ/Quotes Qualifications Experience Contact Info

Featured Article II

A Roadmap for Balanced Productivity Metrics: Are We There Yet?

By Jim Mayes (IT Metrics Strategies, August 2000)

The Balanced Approach

Philosophy

IT outsourcing continues to be a popular course of action for streamlining businesses.  These arrangements always include measurements of performance and productivity improvement.  The questions that we should be asking are:

·     “Why isn’t it just as important to measure performance and productivity related to in-house IT?”

·     “Why don’t we develop in-sourcing contracts, as an alternative to out-sourcing?”

Whether IT is inhouse or outsourced, performance and productivity are important. 

You know how the kids in the back seat are always asking: “Are we there yet?”  Most companies make outsourcing decisions without even knowing their current IT capability – and are often surprised when they find out that they were better off prior to outsourcing.  Similarly, many companies embark on software process improvement programs without knowing their existing level of performance and without any metrics in place to measure productivity improvement.  It’s like the old saying “If you don’t know where you are, a map won’t help.”  This applies to measuring progress as well. It is very difficult to know if we’re “there yet,” if we don’t know where we started and exactly where we still need to go.

Definition of BPM

Balanced Productivity Metrics (BPM) incorporates the use of both quantitative and qualitative data for measuring performance and productivity improvement.  The primary goals of a Software Engineering Institute (SEI) Capability Maturity Model (CMM) based process improvement program are to improve productivity, improve quality, and reduce risk via consistent processes.  BPM focuses on the SEI CMM core measures (size, time, effort, and defects), as well as other data collected to measure process improvement.  A multi-dimensional approach is needed to measure productivity improvement in a way that also provides an understanding of the environmental factors (and other significant factors) that influence project productivity.

The BPM approach is used to measure productivity with regard to software development and maintenance.  BPM is based on the principle that the management of productivity improvement should focus on achieving a balance of time (schedule), cost (effort), and quality (defect rate) [TCQ] 1 improvement.  It is consistent with a balanced scorecard philosophy, since these fundamental metrics components cannot be observed independently with regard to providing a valid productivity assessment.  Balancing TCQ is much like filling a balloon.  If it is filled or compressed too much, it bursts.  If it doesn’t burst and it is compressed in one area, it expands in another.  Likewise, if a project’s schedule (time) is compressed too much, the project will either fail or the project effort (cost), project related defects and production defects (quality) will be disproportionably increased, which increases maintenance effort (cost).  The reverse is also true.  Project schedules can be increased too much, which does reduce effort and defects, but may not provide the desired business value.  Therefore, a balanced measurement approach is needed.  

To quote Karl Wiegers:

A risk with any metrics activity is dysfunctional measurement, in which participants alter their behavior to optimize something that is being measured, rather than focusing on the real organizational goal. For example, if we are measuring output productivity but not quality, expect some developers to change their coding style to expand the volume of material they produce, or to code quickly without regard for bugs. I can write code very fast if it doesn’t actually have to run correctly. The balanced set of measurements helps prevent dysfunctional behavior by monitoring the group’s performance in several complementary aspects of their work that lead to project success.”2

The two primary BPM reporting components for providing a balanced set of metrics, are the:

·     Productivity Metrics Report (PMR), and the

·     Process Performance Report (PPR).

PMR Quantitative Measures

The quantitative measures associated with the Productivity Metrics Report (PMR) are used to assess productivity improvement.  The initial assessment period is considered the baseline year.  Improvement is normally expected at the end of the second year over the baseline year.  Every year thereafter, improvement is expected over the previous year. 

The PMR uses a combination of metrics for calculating the BPM Score, which measures success in achieving productivity improvement goals.  The project metrics are calculated for each software enhancement and new development project, and then averaged for each productivity metrics component.  The maintenance metric is calculated with regard to the yearly hours required to provide ongoing support for the total inventory of supported applications.  The metrics components to be included in calculating the productivity score are shown in Table 1.

Metrics Component

Calculation

Project Schedule Duration

= Total Project Months / Total Projects’ Units of Product Measurement (UPM)

= Months per UPM (Function Points, Lines of Code, etc.)

Project Output Productivity

= Total Project Hours / Total Projects’ Unit of Measure

= Effort Hours per UPM (Could also calculate as FP per hour, etc.)

Project Quality

= Mean Time To Defect (MTTD), for each project is calculated by dividing the number of days exposure during the first 30 days after deployment by the number of defects found

= SUM (Each Project’s MTTD)/ Total Number of Projects

= Weighted Average MTTD

Maintenance Output Productivity

= Total Maintenance Hours / Total Maintenance UPM

= Effort Hours per UPM

Table 1 – BPM Components and Calculations

PPR Qualitative Measures

The qualitative measures associated with the Process Performance Report (PPR), when analyzed along with the quantitative measures, provide insight into the overall process productivity and direction toward achieving process improvement goals.  Looking at the BPM Score – which is a letter grade or number – only provides the current status or contractual measure.  Unless there is an underlying analysis behind that score, there is little information to use for identifying the next steps.  The PPR provides information that can be used to identify problems and areas that have improved, for managing process productivity improvement activities.  This report provides the following analyses, trending, and comparisons to previous period:

·      Demographic data analysis related to organization profiles, project types, size, complexity, and languages

·      Statistical analysis of metrics categories against previous performance trends

·      Analysis of delivery performance, including requirements volatility

·      Significant factors impacting project results

·      Analysis of customer satisfaction survey results in relation to project data

·      Analysis of environmental factors related to tools/methodology, technical difficulty, and personnel

Process Overview

Administration

The BPM process involves joint responsibilities.  The client performance metrics manager (client PMM) manages the process and the database and serves as team lead for a productivity metrics team.  The client PMM will prepare the PPR and the IT metrics group will prepare the PMR, which includes calculating the BPM Score.  Change management with regard to this process will be the responsibility of the productivity metrics team.  (These roles may vary depending on organizational structure.  Additional roles are identified in Table 2.)

Frequency

The PMR should be calculated annually for productivity scoring related to annual software process improvement goals, or contractual agreements in the case of outsourcing.  For purposes of monitoring progress and enabling the parties to promptly address any service or process deficiencies, quarterly BPM Reports containing cumulative data should be created and reviewed.

If, at any time the BPM Score is to be calculated, and one or more metrics or component metrics cannot be calculated, either because the relevant data is not available, or the applicable scoring scale has not been adopted or is not yet in effect, then the BPM Score should be calculated disregarding such metrics.  The weights of the other metrics or component metrics within a given metric shall be increased proportionately such that other metrics or component metrics maintain their respective relative weights.

Data Collection and Dictionary

Data consistency is the key to having a good decision support database; therefore, to insure consistency, the definitions must be documented.  Also, when building an internal historical project repository, the project data points must be categorized.  This will provide stratification such that apples-to-apples project comparisons can be made based upon project type, environment, user organization, development organization, language, etc. The data collection requirements should also be defined and documented.  The collection of software project data during the project is designed to track the status of projects, enable predictive analysis, and facilitate the collection of project closeout data required for the BPM reports.  Upon completion of the project, the data should be summarized in a detailed data collection form.

Tools/Database

BPM project data should be stored in a metrics database.  The project-level information in this database can be used for planning new projects and evaluating risks.  At a minimum, this can be done by using data from similar historical projects for providing an analogy to the project being planned.  This is associated with the CMM Level 2 Software Project Planning key process area and supports the software project estimation and risk management processes. (For more information on this, see my December 2000 ITMS article, “Saving the World One Project at a Time: Planning by the Numbers.”)8

Process Steps

The BPM process-flow schematic is illustrated in Figure 1.  The BPM schematic description and responsibilities are provided in Table 2.  The BPM process begins with collecting baseline project data and the development of a productivity baseline.  The schematic illustrates the next steps after the baseline has been established. 


Figure 1 – BPM Process Flow Schematic

 

Balanced Productivity Metrics Process - Schematic Description

Activity

Performed By

1.   Software Enhancement or New Development Project Launched                      Whenever new projects are launched, it must be determined if they meet the criteria for being included in the productivity metrics process.  There may be some projects that were launched prior to the implementation of this process that should be included as well.  Also, projects should be included that were not initially in scope, but changed such that they subsequently fit the criteria.

IT project manager

2.   Track Project Phase Effort, Schedule, and Defects                                                    The actual effort hours, start date, and end date for each project phase should be tracked.  The project phases include planning, requirements analysis, and main build (detailed design – deployment).  Defects are tracked by severity (1critical, 2 serious, 3 moderate, and 4 tolerable/cosmetic) from the beginning of system test until deployment.  Also, defects must be tracked by severity, for the first 30 days after deployment. 

IT project manager

 

3.   Schedule and Perform Software (UPM) Sizing                                                     Function points, source lines of code or some other method should be used to size the project and application.

IT project manager and technician

4.   Enter Project Data into Detailed Data Collection Form                                               On project completion, project data is entered in the detailed data collection form or spreadsheet.  This form is sent to the IT program manager prior to the project closeout meeting.

IT project manager

 

5.   Review Project Data                                                                                                    The project information provided in the detailed data collection form should be reviewed and validated during the project closeout meeting.  Agreement should be reached on the ratings and results provided.  This can provide an opportunity for joint problem solving related to project impacts and process improvements needed.  The validated detailed data collection form should then be forwarded to the Client performance metrics manager.

Client program manager

 

6.   Enter Detailed Project Data                                                                                         The data contained in the detailed data collection form is entered into the metrics database.

Client performance metrics manager

7.   Track Ongoing Maintenance Effort Hours                                  Actual effort hours charged to ongoing maintenance support must be tracked. 

IT project manager

8.   Schedule and Perform Application (UPM) Baseline Sizing                                        Function points, source lines of code, or some other method should be used to size the applications being maintained.

IT project manager and sizing specialist

9.   Update Maintenance Effort and Size Spreadsheet                                                        A data repository should be maintained containing maintenance productivity data.

IT metrics group

10.  Prepare Quarterly or Yearly BPM Report                                                                   The PMR containing the BPM score, is created by the IT metrics group, and the Performance Manager creates the PPR.

IT metrics group and client performance metrics manager

 Table 2 – BPM Process Schematic Description

Critical Success Factors

The following factors are critical to the success of a Balanced Productivity Metrics Program:

bulletClient and IT partnering relationship with open communication paths
bulletSenior management support and a documented policy statement
bulletClient program management, client performance metrics management, IT project management, and IT metrics group support, and synchronized processes
bulletAccurate data collection and sizing on a timely basis
bulletUsing project-level quantitative and qualitative metrics data for software project planning and process improvement (i.e., don’t just accumulate it for contractual purposes)
bullet Communication, training, and mentoring related to the BPM process

Scoring

Metrics Component Scores

Each BPM Metric Component will be given a letter score determined in accordance with the scoring scale shown in Table 3, assuming a 5% yearly improvement for each component is desired.  This letter grade will be converted to a numerical score using the scale also shown in Table 3.  (The percentages and letter grades can be adjusted based upon the yearly improvement desired for each TCQ category.)           

 

Grade

Numerical Score

Component Grade (calculated for each Metrics Component and then converted to numerical score)

A

5.0

A for = 5.5% or better (exceeded)

B

4.0

B for = 4.5% to 5.4% (expected)

C

3.0

C for = 0 to 4.4%

D

2.0

D for = <0 to –4.4%

F

0.0

F for = -4.5% or worse

Table 3 (T3) – Metrics Category Component Grading

Metrics Component Weighting and Balanced Scoring

Business value should determine the weightings with regard to Project TCQ and Maintenance, since this is a business decision as to which component has the highest priority, or if they are all equally important.  Of course this will vary from project to project, depending on the circumstances, therefore the BPM weightings should reflect the overall philosophy for the organization.  Component/Category weightings should be established once per year, at the beginning of each annual measurement cycle.  If everything were equally weighted, the total BPM composite numerical score would be calculated as shown it Table 4, and the final composite BPM grade would be calculated as shown in Table 5.  Examples of BPM Score Calculations are provided in Tables 6, 7, and 8.

 

Component

Weight

Composite Numerical Score Calculation

Project schedule duration

25%

Project schedule numerical score (T3) x 0.25 (T4)

Project output productivity

25%

+ Project output numerical score (T3) x 0.25 (T4)

Project quality

25%

+ Project quality numerical score (T3) x 0.25 (T4)

Maintenance output productivity

25%

+ Maintenance output numerical score (T3) x 0.25(T4)

Total

100%

= Composite numerical score (convert to grade in T5)

 Table 4 (T4) – Numerical Component Weighting and Composite Calculation

 

Grade

Numerical Score

Assessment

A

4.5 – 5.0

Exceeds standards

B

4.0 – 4.4

Meets standards

C

3.0 – 3.9

Does not meet standards

D

2.0 – 2.9

Does not meet standards

F

0.0 – 1.9

Does not meet standards

 Table 5 (T5) – Final BPM Composite Grading

 

Metrics Component

Baseline Values

Year 2 Values

% Change

Component Grade (T3)

Numerical Score (T3) x Weight (T4)

Composite Score and Grade (T5)

Project Schedule Duration

1.1

1.045

5% better

B

4 x .25

1

Project Output Productivity

3.6

3.42

5% better

B

4 x .25

1

Project Quality

4.5

4.73

5% better

B

4 x .25

1

Maintenance Output Productivity

1.8

1.71

5% better

B

4 x .25

1

Total

 

 

 

 

 

4 = B (meets standards)

 Table 6 – Example 1: BPM Score Calculation

 

Metrics Component

Baseline Values

Year 2 Values

% Change

Component Grade

Component Score x Weight

Composite Scoring

Project Schedule Duration

1.1

1.045

5% better

B

4 x .25

1

Project Output Productivity

3.6

3.49

3% better

C

3 x .25

0.75

Project Quality

4.5

4.73

5% better

B

4 x .25

1

Maintenance Output Productivity

1.8

1.71

5% better

B

4 x .25

1

Total

 

 

 

 

 

3.75 = C (does not meet standards)

 Table 7 – Example 2: BPM Score Calculation

 

Metrics Component

Baseline Values

Year 2 Values

% Change

Component Grade

Component Score x Weight

Composite Scoring

Project Schedule Duration

1.1

1.045

5% better

B

4 x .25

1

Project Output Productivity

3.6

3.39

6% better

A

5 x .25

1.25

Project Quality

4.5

4.73

5% better

B

4 x .25

1

Maintenance Output Productivity

1.8

1.75

3% better

C

3 x .25

0.75

Total

 

 

 

 

 

4 = B (Meets Standards)

Table 8 – Example 3: BPM Score Calculation

Conclusion

The measurement of productivity and performance is just as important for inhouse IT as it is for an outsourced IT environment.  A balanced approach, as provided by the BPM process, is needed in order to ensure that all aspects of productivity are assessed and that the correct behavior with regard to performance is achieved.  Some outsourcing vendors will argue that only one measure should be used, such as effort per unit of output, but to most clients, schedule and quality are equally important.  Therefore, measures should be in place such that schedule or quality cannot be sacrificed, and to prevent dysfunctional measurement behavior from occurring.  It is beneficial to IT that all factors be measured, due to the clients’ ever changing TCQ priorities.  In addition to balancing the TCQ metrics and quantitatively tracking projects, it is also important to understand the relationship between the quantitative and qualitative measures that influence productivity.  When we trend the quantitative and qualitative data and compare the results to previous internal benchmarks, our corporate knowledge will be increased such that we can make intelligent business decisions. The BPM process can help us to understand the factors that influence our productivity, know where we are on the map, effectively predict where we are headed, and recognize when we get there.

References

1.   Null, Christopher. (2001, November). “Mess with The Bull, Get the Horns.” Smart Business.

2.   Mayes, Jim. (2001, August). “A Road Map for Balanced Productivity Metrics: Are We There Yet?” IT Metrics Strategies. Cutter Consortium.

3.   Paulk, Weber, Garcia, Chrissis, Bush. (1993). Key Practices of the Capability Maturity Model, Version 1.1. CMU/SEI-93-TR-25, ESC-TR-93-178.

4.   Wiegers, Karl. (1999, July). “A Software Metrics Primer.” Software Development

5.   Mayes, Jim. (2000, March). “Achieving Business Objectives: Balancing Time, Cost, and Quality.” IT Metrics Strategies. Cutter Consortium.

6.   Lombardi, Vince, former NFL coach. Green Bay Packers.

7.   Wheeler, Donald, (PhD.) Alabama School of Mathematics and Sciences

8.   Mayes, Jim. (2001, August). “Saving the World, One Project at a Time: Planning by the Numbers.” IT Metrics Strategies. Cutter Consortium.

9.   Baumert and McKinney. (1992) Software Measures and the Capability Maturity Model. CMU/SEI-92-TR-25, Technical Report. 

10.  Florac, Park, and Carleton. (1997) Practical Software Measurement: Measuring for Process Management and Improvement. CMU/SEI-97-HB-003.

 

 

Mayes Consulting, LLC

Quantitative Software Engineering  
Copyright (c) 2001 Mayes Consulting
1605 Kinsmon Lane
Marietta, GA  30062
770-649-8599 or 404-754-2707
jimmayes@bellsouth.net