» Metrics
» Attributes of Good Metrics
» Choosing a Metric
» Software Metrics
» Type of Metrics

Metrics

Metric is a measure for quantitatively assessing, controlling or selecting a person, process, event, or institution, along with the procedures to carry out measurements and the procedures for the interpretation of the assessment in the light of previous or comparable assessments. Metrics are usually specialized by the subject area, in which case they are valid only within a certain domain and cannot be directly benchmarked or interpreted outside it. This factor severely limits the applicability of metrics, for instance in comparing performance across domains. The prestige attached to them may be said to relate to a ‘quantifiability fallacy’, the erroneous belief that if a conclusion is reached by quantitative measurement, it must be vindicated, irrespective of what parameters or purpose the investigation is supposed to have.

WHY METRICS?

Before we consider software measurement and collecting software metrics data, we need to ask ourselves: why are we doing this? Do organizations need data on their software projects and management? Does the software industry need data about itself? There is management pressure to build systems faster, better and at minimum cost. The return on investment that an organization can get from the money it spends on software development has come under increased scrutiny from senior business executives and directors. Consequently software development projects now have to operate like other parts of the organization, being aware of its performance and its contribution to the organization’s success and opportunities for improvement. How can a Manager achieve this without performance data? The below quotes by Lord Kelvin and Tom Demarco make us understand the importance of measurement.

“When you can measure what you are speaking about and express it in numbers, You know something about it, but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a measure and unsatisfactory kind: it may be the beginnings of knowledge but you have scarcely in your thoughts advanced to the stag of science.” _Lord Kelvin

“You cannot control what you cannot measure.” _Tom De Marco

So what is it that managers need to know? Here are some of the questions that the Managers need to know in their project:

• How do I know if my internal operation is performing satisfactorily?

• How do I decide whether I should outsource some or all of my operations?

• How do I know if my outsourcer is performing?

• What are the risk factors I should consider in a project?

• What questions should I ask to ensure that a project proposal is realistic?

• How do I know if a project is healthy? i.e. what should I be worrying about?

None of these questions can be answered without sound software metric system in place.

A Properly established metric helps achieve missions, visions, goals and objectives. Measurement data is most reliable when it is generated as a byproduct of producing a product or service. Metric can be used to gauge the status, effectiveness and efficiency of processes, customer satisfaction, product quality and as a tool for management concepts, the use of metrics in a software development environment, variation, process capability, risk management, the ways measurement can be used and how to implement an effective metrics program.

Measurement Concepts

To effectively measure, one needs to know the basic concepts of measurement. This section provides those basic measurement concepts.

Standard Units of Measure

A measure is a single quantitative attribute of an entity. It is the basic building block for a measurement program. Examples of measures are lines of code (LOC) , work effort, or number of defects. Since quantitative measures should be expressed in numbers. For example, the measure LOC refers to the “number” of lines and work effort refers to the “number” of hours, days, or months.

Measurement cannot be used effectively until the standard units of measure have been defined. For example, talking about lines of code does not make sense until the measure LOC has been defined. Lines of code may mean LOC written, executable LOC written, or non-compound LOC written. If a line of code contained a compound statement (Such as a nested IF statement two levels deep) it could be counted as one or two lines of code. Additionally, organizations may use weighting factors: for example, one verb would weighted as more complete than other verbs in the same programming language expressed in numbers, For example, the measure LOC refers to the “number” of lines and work effort refers to the “number” of hours, days, or months.

Measurement cannot be used effectively unit the standard units of measure have been defined .Lines of code may mean LOC written, executable LOC written, or non-compound LOC written. If a line of code counted a compound statement (Such as a nested IF statement two levels deep ) it could be counted as one or two lines of code. Additionally, organizations may use weighing factors; for example, one verb would be weighted as more complete than other verbs in the same programming language.

Standard units of measure are the base on which all measurement exists. Measurement programs typically have between five and fifty standard units.

A metric is a derived (Calculated or composite) unit of measurement that cannot be directly observed, but is created by combing or relating two or more measures. A metric normalizes data so that comparison is possible. Since metrics are combinations of measures they can add more value in understanding or evaluating a process than plain measures. Examples of metrics are mean time to failure and actual effort compared to estimated effort.

Objective and Subjective Measurement

Objective measurement uses hard data that can be obtained by counting, staking, weighing, timing, etc. Examples include number of defects, hours worked, or completed deliverables. An objective measurement should result in identical values for a given measure, when measured by two or more qualified observers.

Subjective data is normally observed or perceived. It is a person’s perception of a product or activity, and includes personal attitudes, feelings and opinions, such as how easy a system is to use, or the skill level needed to executive the system. With subjective measurement, even qualified observers may determine different value for a given measure, since their subjective judgment is involved in arriving at the measured value. The reliability of subjective measurement can be improved through the use of guidelines, which define the characteristics that make the measurement result one value or another.

Objective measurement is more reliable than subjective measurement, but as a general rule, subjective measurement is considered more important. The more difficult something is to measure, the more valuable it is. For example, it is more to know how effective a person is in performing a job (subjective measurement), than knowing they got to work on time (objective measurement). Following are a few other examples of objective and subjective measures:

The size of a software program measured in LOC is an objective product measure. Any informed person, working from the same definition of LOC, should obtain the same measure value for a given program.

The classification of software as user-friendly is a subjective product measure. For a scale of 1-5, customers of the software would likely rate the product differently. The reliability of the measure could be improved by providing customers with a guideline that describes how having or not having a particular attribute affects the scale.

Development time is an objective process measure. Level of programmer experience is a subjective process measure.

Types of Measurement Data

Before measurement data is collected and used, the type of information involved must be considered. It should be collected for a specific purpose. Usually the data is used in a process model, used in other calculations, or is subjected to statistical analyses. Statisticians recognize four types of measured data, which are summarized in table below.

Data Type		Possible Operations		Description of Data
Normal		= ?		Categories
Ordinal		< >		Rankings
Interval		+		Differences
Ratio		/		Absolute Zero

Nominal Data

This data can be categorized. For example, a program can be classified as database software, operating system, etc. Normal data cannot be subjected to arithmetic operations of any type, and the values cannot be ranked in any “natural order”. The only possible operation is to determine whether something is the same type as something else. Nominal data can be objective, depending on the rules for classification.

Ordinal Data

This data can be ranked, but differences or ratios between values are not meaningful. For example, programmer experience level may be measured as low, medium, or high. For ordinal data to be used in an objective measurement the criteria for placement in the various categories must be well defined; otherwise, it is subjective.

Interval Data

This data can be ranked and can exhibit meaningful differences between values. Interval data has no absolute zero, and ratios of values are not necessarily meaningful. For example, a program with a complexity value of 6 is four units more complex than a program with a complexity of 2, but it is probably not meaningful to say that the first program is three times as complex as the second. T.J. McCabe’s complexity metric is an example of an interval scale.

Ratio Data

This data has an absolute Zero and meaningful ratios can be calculated. Measuring program size by LOC is an example. A program of 2,000 lines can be considered twice as large as a program of 1,000 lines. It is important to understand the measurement scale associated with a given measure or metric. Many proposed measurements use values from an interval, ordinal, or nominal scale. If the values are to be used in mathematical equations designed to represent a model of the software process, measurements associated with a ratio scale are performed, since the ratio scale allows mathematical operations to be meaningfully applied.

Measures of Central Tendency

The measures of central tendency are the mean, medium and mode. The mean is the average of the items in the population; the medium is the item at which half the items in the population are blow this items and half the items are above this item; and the mode represents which items are repeated most frequently.

For example, if a population of numbers are: 1, 2, 2, 3, 4, 5, and 11:

The mean is “4” because 1=2=2=3=4=5=11=28 and 28 %7=4.

The medium is “3” because there are three values less and three values higher than 3.

The mode is “2” because that is the item with the most occurrences.

Attributes of Good Metrics

Ideally models should be developed that are capable of predicting process or product parameters, not just describing them. This is facilitated by measures and resulting metrics that are;

• Simple and precisely definable, so it is clear how can be evaluated

• Objective

• Easily obtainable at reasonable cost

• Valid, measuring what they are intended to measure

• Robust, being relatively insensitive to intuitively small changes in the process or product

Before being approved for use, measures and metrics should be subjected to the six tests described below.

Reliability

This test refers to the consistency of measurement. If taken by two people, the same results should be obtained. Sometimes measures are unreliable because of the measurement technique. For example, human error could make counting LOC unreliable ,but the use of an automated code analyzer would result in the same answer ach time it is run against an unchanged program.

Validity

This test indicates the degree to which a measure actually measure actually measures what it was intended to measure. If actual project work effort is intended to qualify the total time spent on a software development project, but overtime or time spent on the project by those outside the project team is not included, the measure is invalid for its intended purpose. A measure can be reliable, but invalid. An unreliable measure cannot be valid.

Ease of Use and Simplicity

These two tests are functions of how easy it is to capture and use the measurement data.

Timeliness

This test refers to whether the information can be reported in sufficient time to impact the decisions needed to manage effectively.

Calibration

This test indicates the modification of a measurement so it becomes more valid; for example, modifying a customer survey to better reflect the true opinions of the customer.

Using Quantitative Data to Manage an IT Function

An integral part of an IT function is quantitative management, which contains two aspects; measurement dashboards and statistical process control.

Choosing a Metric

To select suitable metrics for an organization, first have a look and organizational goals. From each organizational goal derive a set of questions that needs to be answered to achieve the particular goals. The answers of these questions will provide you a set of metrics that the organization should focus upon. The measurement goal of the metric is defined based on the process Area and the Specific/ Generic practice. It is seen that this goal relates to the operational metric(s), which converges to a set of managerial metric(s), which in turn leads to a set of Business metrics.

METRIC SET

The defined indicators are consistent with the Software Engineering Institute’s Capability Maturity Model (CMMI) .The blow table shows the indicator categories, the management insight provided, and the specific indicators for recommended metrics. Depending upon the nature of the project, specific contractual requirements, or management preference, a project may choose to collect additional metrics or to tailor the recommended set.

Recommended Metric Set for a project

Indicator Category		Management Insight		Indicators
Progress		Provides information on how well the project is performing with respect to its schedule.		Actual vs. planned task completions. Actual vs. planned durations.
Efforts		Provides visibility into the contributions of staffing on project costs, schedule adherence, and product quality.		Actual vs. planned staffing profiles.
Cost		Provides visibility into the contributions of staffing on project costs, schedule adherence, and product quality.		Actual vs. planned costs Cost and schedule variances.
Review Results		Provides status of action items from life-cycle review.		Status of action items.
Trouble Reports		Provides insight into product and process quality and the effectiveness of the testing.		Status of trouble reports Number of trouble reports opened, closed, etc. during reporting period.
Requirements Stability		Provides visibility into the magnitude and impact of requirements changes.		Number of requirements changes/ clarifications. Distribution of requirements over releases.
Size Stability		Provides insight into the completeness and stability of the requirements and into the ability of the staff to complete the project within the current budget and schedule.		Size growth distribution of size over releases.
Training		Providing information on  the training program and staff.		Actual vs. planned number of personnel attend.

Software Metrics

Software metrics are numerical data related to software development. Metrics strongly support software project management activities. They relate to the four functions of management as follows:

1. Planning – Metrics serve as a basis of cost estimating, training planning, resource planning, scheduling, and budgeting.

2. Organizing - Size and schedule metrics influence a project’s organization.

3. Controlling - Metrics are used to status and track software development activities for compliance to plans.

4. Improving - Metrics are used as a tool for process improvement and to identify where improvement efforts should be concentrated and measure the effects of process improvement efforts.

A metric quantifies a characteristic of a process or product. Metrics can be directly observable quantities. Examples of raw metrics include the number of source lines of code, number of documentation pages, number of staff-hours, number of tests, number of requirements etc. Examples of derived metrics include source lines of code per staff-hour, defects per thousand lines of code, or a cost performance index.

The use of measurement in the software life cycle requires the development and use of software metrics, which are standardized through the use of defined units of measure. This measurement program enables management to control and manage software throughout its entire life.

But software measurement is a challenging but essential component of a healthy and highly capable software engineering culture. Effective project management mandates measurement of the capability of product development process, and raising alarms at distinct points in the software development life cycle (SDLC) to take corrective actions. Software measures should help ensure and predict output quality, and drive deliveries that meet planned schedules and are within budget.

Software metrics programs not only improve software quality but also reduce overall costs. A SDLC model with regular reviews, controls and a proper software metrics program at very phase may demand a higher investment in the initial stages, but the overall cost is significantly less because of lower defect rates. The converse is also true.

Both the software product and the process by which it is developed can be measured. The software product should be viewed as an abstract object that evolves from an initial statement of need to a finished software system, including source and object code and the various forms of documentation produced during development process. These metrics and models are used to estimate and predict product costs and schedules, and to measure productivity and product quality. Information gained from the measurements and models can be used in the management and control of the development process, leading to improved results.

Software metrics are typically used for estimation (size, effort and cost), productivity measurements, reliability measurements, quality modeling, and project & task management. A metric quantifies a characteristic of a process or product and defines what is to measured. They help in controlling software projects and learning more about the way organization works.

Current measures of performance need to be used to define a baseline of performance: otherwise the targets will just b speculations. For example;

• Projects status cannot be tracked meaningfully unless the actual effort and time spent on each task are compared to planed effort and time respectively.

• It is not possible to decide whether a product is stable enough to ship unless one tracks the rate at which the development team is finding and fixing defects.

Type of Metrics

Metrics can be classified into various categories depending up how they are measured or calculated such as

• Direct measurement
0 number of line of code

• Indirect/ derived measurement
0 defect density =no. of defects in a software
O product/ total size of product

• Prediction
0 predict effort required to develop software

• Measure of the functionality
0 function point count

This can be further classified into depending on the characteristics like

• Product metrics
0 size metrics
0 complexity metrics

• Quality metrics
• Process metrics
• Resource metrics
• Project metrics

All the metrics are typically grouped into the following categories:

• Operational Metrics that help the project manager to take appropriate decisions to keep the project in control.

• Managerial Metrics that are used to help in senior management decisions.

• Business Metrics related to cost, schedules and customer related measures. The categories are strongly linked and impact one another. For example:

• If managerial metrics are not controlled they start impacting business metrics; it may result in loss of revenue, loss of business and reduced customer satisfaction.

• If operational metrics exceed the threshold values, they start reflecting in the managerial metrics.

A business metric may have a number of managerial metrics that in turn will have a set of operational metrics.

Measurement Dashboards

Measurement dashboards (also called key indicators) are used to monitor progress and initiate change. A measurement dashboard is analogous to the dashboard on a car. Various key indicators are presented in comparison to their own desired target value (i.e., speed, oil temperature). Arrayed together, they provide an overall snapshot of the car’s performance, health, and operating quality. Measurement dashboards help ensure that all critical performance areas are analyzed in relation to other areas. Indicators evaluated alone can lead to faulty conclusions and decision – making.

Measurable results defined by an organization can be organized into dashboards for different levels of management. Line managers use process or tactical dashboards to manage the process. Senior management uses strategic dashboards to manage the function or organization and track to mission, vision, or goals, Using dashboards is known as “management by fact”.

Key Indicators

Key indicators are the metrics used by management to help them fulfill their management responsibilities. Managers decide what key metrics they need. A dashboard is comprised of the total number of key indicators used by a single manager. The concept of “key” means the metric is considered important by the user in managing job responsibilities. Normally the key indicators are created from many different measures.

Implementing a Measurement program

The key to a good measurement program is to know what results are wanted, and what drives those results. Then metrics need to be developed to measure those results and drivers. This section explains how an effective measurement program is implemented.

The Need for Measurement

Current IT management is often ineffective because the IT function is extremely complex, and has few well-defined, reliable process or product measures to guide and evaluate results. Thus, accurate and effective estimating, planning, and control are nearly impossible to achieve. Projects are often characterized by:

• Schedule and cost estimates that are grossly inaccurate
• Poor quality software
• A productivity rate is increasing more slowly than the demand for software
• Customer dissatisfaction

Addressing these problems requires more accurate schedule and cost estimates, better- quality products, and higher productivity that can be achieved through improved software management. Improvement of the management process depends upon improved ability to identify, measure, and control essential parameters of the IT processes. Measurement is an algorithm connecting the desired result (i.e., the effect wanted) with the contributors or causes that will enable that effect to be achieved. The results are what management wants, and the contributors are attributes of the processes that will be used to achieve those results. By measuring processes and products, information is obtained that helps control schedule, cost quality, productivity, and customer satisfaction. Consistent measurements provide data for the following:

• Expressing requirements, objectives, and acceptance criteria in a quantitative manner
• Monitoring progress of a project and/or product
• Making tradeoffs in the allocation of resources
• Evaluating process and product quality
• Anticipating problems
• Predicting deadlines of current project
• Estimating future projects of a similar nature

Results indicate that implementation and application of a measurement program can help achieve better management results, both in the short run (for a given project ) and in the long run (improving productivity on future projects).

Prerequisites

Implementing a measurement program requires four prerequisite steps:

1. Perceive the need for a measurement program and make a commitment to it.

The lack of timely and usable quantitative information to solve major project problems becomes apparent at the senior management level. Seeing the need for better management information, the site manager, general manager, or division VP sponsors an organizational commitment to a understanding at all levels in the organization.

2. Identify a champion and change agent, and assign organizational responsibility.

A champion is an advocate for the program by virtue of his/her technical credibility and influence. Champions are managers at the senior, middle, program, or project level, and are assisted by a change agent.

A change agent is a management leader empowered by the sponsor and champions to plan and implement the program. Change agents are most effective when they are members of working groups that will benefit from the measurement program. They have the operational knowledge needed to schedule, monitor, control, and report the accomplishments of the measurement program.

The project or organization selected for the implementation of the measurement program should have the resources, authority, and responsibility to make the program happen. Unless a very limited program is contemplated, responsibility for implementing the program should not be given to a single individual.

During this step, idea generators, idea exploiters, and information gatekeepers should be identified. Idea generators contribute new ideas about the measurement program. Ida exploiters implement the new ideas in the form of pragmatic programs. Information gatekeepers are experts in measurement, and can provide informed realities of it. These people implement the ideas to form a workable measurement program and ensure developers accept the program.

3. Establish tangible objectives and meaningful measurement program activities.

The change agent guides the planning of the program, including the creation of program objectives and the design of program activities. The planning takes the sponsor’s goals for more effective information and defines expected results, needed resources, tasks, and organizations responsible for implementing the program.

4. Facilitate management buy-in at all levels for the measurement program.

The measurement program’s sponsor must clearly inform all levels of management of his/her interest in the measurement program and motive their cooperation. They need to know the implementation team’s goals, responsibilities, authority, and interfaces with other organizations.

Also important is to work with affected managers to obtain their buy-in, by tailoring the implementation so that most of their needs are met.

Conclusion

A properly established measurement system is used to help achieve missions, visions, goals, and objectives. Measurement data is most reliable when it is generated as a by-product of producing a product or service. The QA analyst must ensure that quantitative data is valued and reliable, and presented to management in a timely and easy-to-use manner. Measurement can be used to gauge the status, effectiveness and efficiency of processes, customer satisfaction, product quality, and as a tool for management to use in their decision-making processes. This unit explains the measurement concepts, the us of measurement in a software development environment, variation, process capability, risk management, the ways measurement can be used and how to implement an effective measurement program.