Security Metrics Development and Implementation Based on NIST Directives

Date: Jan 17, 2011

Return to the article

Metrics are tools that should be used to aid in decision making, and improve performance and accountability through collection, analysis, and reporting of relevant performance-related data.

Security metrics are based on security performance goals and objectives. Security performance goals state the desired results of implementation of a security program. Security performance objectives, in turn, enable the accomplishment of goals. They do this by identifying practices defined by policies, standards and procedures that direct consistent implementation of data protection controls across the organization.

The policies, standards, and procedures describe the controls (technology, process, administrative) that should be in place, and metrics provide insight into the implementation, efficiency, effectiveness, and business impact of these controls. Before beginning the process of developing a security metric program, an organization first needs to get the proper policies, standards, and procedures developed and in place—otherwise there is nothing to use as benchmarks.

Security metrics monitor the accomplishment of the goals and objectives outlined in the stated documents. They accomplish this by quantifying the level of implementation of the security controls and the effectiveness and efficiency of the controls, analyzing the adequacy of security activities, and identifying possible improvement actions.

The following matters must be considered during development and implementation of a security metrics program:

• Metrics must yield quantifiable information (percentages, averages, and numbers).
• The data that supports the metrics needs to be readily obtainable.
• Only repeatable processes should be considered for measurement.
• Metrics must be useful for tracking performance and directing resources.

The metrics development process, as described below, ensures that metrics are developed with the purpose of identifying causes of poor performance, and that they therefore point to appropriate corrective actions.

Types

An organization should develop and collect metrics of three types:

• Implementation metrics to measure implementation of security controls
• Effectiveness/efficiency metrics to measure the results of security controls
• Impact metrics to measure the impact on business or mission of security events

The types of metrics that can realistically be obtained and are useful for performance improvement depend on the maturity of the organization’s security program. Although different types of metrics can be used simultaneously, the primary focus of security metrics shifts as the implementation of security controls matures.

It cannot be emphasized enough that great diligence must be taken when developing initial metrics. Capturing the wrong type of data ends up in a waste of time and resources. Capturing partial data shows only part of the story. And capturing data that does not have supportive evidence provides a false sense of security.

Overview of Metrics Program

A security metrics program within an organization should include four interdependent components, as illustrated in the figure below.

The first component, a foundation of strong upper-level management support, is critical. It is crucial not only for the success of the security program, but also for the implementation of a security metrics program. This support establishes a focus on security within the highest levels of the organization. A solid foundation means the proactive support of persons in positions of control of security resources. Without such as foundation, the effectiveness of the security metrics program can fail when pressured by politics and budget limitations.

The proper development of a metric program requires both people with the correct skill set and funds to support the efforts. Many times upper management will ask for performance data but not realize how much work has to go into gathering this data in a quantifiable manner.

The second component of an effective program is practical security policies, standards and procedures backed by the authority to enforce compliance. Practical security policies, standards, and procedures are those that are attainable and that provide meaningful security through appropriate controls. Metrics are not easily obtainable if there are no policies, standards, and procedures in place.

The third component is developing and establishing quantifiable performance metrics that are designed to capture and provide meaningful performance data. Quantifiable security metrics must be based on security performance goals and objectives, be easily obtainable, and be feasible to measure. They must also be repeatable, provide relevant performance trends over time, and be useful for tracking performance and directing resources to the appropriate places.

Finally, the security metrics program itself must emphasize consistent and periodic analysis of the metrics data. The results of this analysis are used to apply lessons learned, improve the effectiveness of existing security controls, and plan future controls to meet new security requirements as they occur. Accurate data collection must be a priority with stakeholders and users if the collected data is to be meaningful to the management and improvement of protection of sensitive data.

The success of implementing a security metrics program should be judged by the meaningful results that are produced. A comprehensive security metrics analysis program should provide substantive justification for decisions that directly affect the security posture of an organization. These decisions include budget and personnel requests and allocation of available resources. A security metrics program also should provide a precise basis for preparation of required security performance-related reports.

Purpose, Approach, and Objectives

The purpose of measuring performance is to monitor the status of measured activities and to assist improvement in those activities by applying corrective actions, based on observed measurements.

Security metrics can be obtained at different levels within an organization. Detailed metrics, collected at the system level, can be aggregated and rolled up to progressively higher levels, depending on the size and complexity of an organization.

Security performance objectives enable the accomplishment of goals by identifying practices defined by security policies, standards and procedures that direct consistent implementation of security controls across the organization.

These goals and objectives are to be outlined in policies, standards, and procedures and metrics are then to be built to measure the effectiveness of the controls that are in place to accomplish these goals and objectives.

Requirements

Some security metrics monitor measurable data that may be difficult to obtain. Metrics must use easily obtainable data to ensure that the burden of measurement on an organization does not absorb too many resources. Only processes that can be consistent, repeatable, and measurable should be considered for measurement.

At this point some processes within the security program may not be consistent and repeatable enough to be properly measured. In many cases it is critical to first get processes defined and matured and then measure their success factors.

To track performance and assist in directing resources, metrics must provide relevant performance trends over time and point to actions aimed at alleviating problems. Management should use metrics to assess performance by reviewing metrics trends, identifying and prioritizing corrective actions, and directing the application of those corrective actions based on risk mitigation factors and available resources.

The metrics development process ensures that metrics are developed for identifying causes of poor performance. They therefore point to appropriate corrective actions; accomplishment of goals and objectives by quantifying implementation of the security controls and the effectiveness and efficiency of the controls, analyzing the adequacy of security activities, and identifying possible improvement actions.

Security metrics must yield quantifiable information for comparison, apply formulas for analysis, and track changes using the same points of reference. Percentages or averages are most common, although absolute numbers are sometimes useful, depending on the activity that is being measured.

Benefits of Using Metrics

A security metrics program can provide an organization with a number of organizational and financial benefits. An organization can improve security accountability by deploying security metrics. The process of data collection and reporting will enable the management to pinpoint specific technical, operational, or management controls that are not being used or are used incorrectly. Security metrics can be created to measure each aspect of an organization’s security. For example, the results of risk assessments, penetration testing, security testing and evaluation, and other security-related activities can be quantified and used as data sources for metrics. Using the results of the metrics analysis, program managers and system owners can isolate problems, use collected data to justify investment requests, and then target investments specifically to the areas needing improvement. Use of security metrics will allow an organization to measure successes and failures of past and current security investments. Metrics will provide quantifiable data that will support allocation of resources for future investments. By using metrics to target security investments, an organization can get the best value from available resources.

Security metrics can also help determine the effectiveness of implemented security processes, procedures, and controls. They do this by relating the results of security activities such as incident data and revenue lost to cyber attacks to respective security requirements and security investments.

It is important to realize that the development of these types of metrics happens over time, sometimes years. Metric development goes through an evolutionary process and is directly related to the maturity of the security program and the resources allocated to it. Incremental benefits will be achieved as the metrics are developed and mature over time.

Metrics Types

The maturity of an organization’s security program determines the type of metrics that can be gathered successfully as shown the figure below.

A security program’s maturity is defined by the existence and institutionalization of its processes and procedures. As a security program matures, its policies become more detailed and better documented, the processes that it uses become more standardized and institutionalized, and it produces data that can be used for performance measurement in greater quantity and enhanced quality.

The security program progresses from: having policies (Level 1); to having detailed procedures (Level 2); to implementing these procedures (Level 3); to testing compliance with, and effectiveness of, the procedures (Level 4); and, finally, fully integrating policies and procedures into daily operations (Level 5).

A mature program normally deploys multiple tracking mechanisms to document and quantify various aspects of program performance. As more data becomes available, the difficulty of measurement decreases, and the ability to automate data collection increases. Manual data collection involves developing questionnaires and conducting interviews and surveys with the organization’s personnel. Automated data collection depends on the availability of data from automated sources, as opposed to data available from personnel.

Typically, more useful data will become available as a security program matures from semi-automated and automated data sources, such as self-assessment tools, security event management, incident reporting, and response databases. Metrics data collection is fully automated when all data is gathered by using automated data sources without human involvement or intervention.

The types of metrics—implementation, efficiency and effectiveness, and impact—that can realistically be obtained and that is useful for performance improvement depend upon the maturity of the implementation of the security controls. Although different types of metrics can be used simultaneously, the primary focus of metrics shifts as the implementation of security controls matures. When security controls have been defined in procedures and are being implemented, the primary focus will be on the level of implementation. When a system progresses through Level 1 and Level 2, the results of these metrics will be less than 100 percent, and the system will have not yet reached Level 3. When the results reach and remain at 100 percent, the system has fully implemented security controls and has reached Level 3.

As security controls are documented and implemented, the ability to reliably collect the outcome of their implementation improves. As an organization’s security program evolves and performance data becomes more readily available, metrics will focus on program efficiency—the timeliness of security service delivery and effectiveness—and the operational results of security control implementation. Once security is integrated into an organization’s processes, the processes become self-regenerating, measurement data collection becomes fully automated, and data correlation analysis can determine the mission or business impact of security-related actions and events.

The metrics at Level 4 and Level 5 concentrate on measuring effectiveness and efficiency of implemented security controls, as well as the impact of these controls on an organization’s mission. These metrics focus on the evidence and results of testing and integration. Instead of measuring the percentage of approved security policies and procedures, these metrics concentrate on validating whether security controls, described in the security policies and procedures, are effective in protecting an organization’s sensitive data. For example, computing the percentage of crackable passwords within a predefined time threshold will validate the effectiveness of an organization’s password policy, by measuring the length of time required to break policy-compliant passwords. The impact metrics would quantify incidents by type (e.g., root compromise, password compromise, malicious code, denial of service).

It is not feasible to develop Level 4 and 5 metrics for security program if it is really at a Level 1 or 2 in its maturity progress. As the security program matures, these higher level metrics that deal with effectiveness and efficiency can be developed and used.

Data Management Concerns

The importance of standardizing reporting processes cannot be overemphasized. To ascertain the quality and validity of data, data collection methods and data repositories used either directly or as data sources for metrics data collection and reporting should be standardized. The validity of data is suspect if the primary data source is an incident-reporting database that stores only the information reported by some organizational elements, or if reporting processes between organizations are inconsistent. When an organization is developing and implementing processes that may serve as inputs to its security metrics program, it must ensure that data gathering and reporting are clearly defined.

The organization’s personnel must understand that although they may collect a lot of security data, not all data will be useful for their metrics program at any given point in time. Any data collection, specifically for the purpose of security metrics, must be as non-intrusive as possible. It also must be of maximum usefulness. This is to ensure that available resources are primarily used to correct problems, and not to collect data for data’s sake.

The establishment of a metrics program will require a substantial investment to ensure that the program is properly implemented and to maximize its benefits. The resources required for maintaining the program are not expected to be as significant.

Two processes guide the establishment and operation of a security metrics program: metrics development and metrics implementation. The metrics development process establishes the initial set of metrics and selection of the metrics subset appropriate for an organization at a given time. The metrics implementation process operates a metrics program that is iterative, and that ensures appropriate aspects of security are measured for a specific time. Security metrics can be used to progressively measure the implementation, efficiency, effectiveness, and the business impact of security activities within an organization or for specific systems.

The metrics development process is illustrated in the figure below.

The security metrics development process consists of two major activities:

1. Identification and definition of the current security program security controls
2. Development and selection of specific metrics to measure implementation, efficiency, effectiveness, and impact of the security controls

Stakeholder Interest Identification

In Phase 1 of the metrics development process shown in Figure 3, anyone within an organization should be a security stakeholder. Some functions, however, will have a greater stake in security than others. A stakeholder is someone who is responsible for some aspect of security, thus should be directly or indirectly involved with metric development, implementation and reporting.

Examples of primary security stakeholders are:

• Executive Management
• Chief Security Officer (CSO)
• Chief Information Officer (CIO)
• Program Manager/System Owner
• System Administrator/Network Administrator
• IT Support Personnel

The secondary security stakeholders do not have security as their primary mission but for some aspects of their operations.

Examples of secondary security stakeholders include:

• Chief Financial Officer (CFO)
• Training Organization
• Human Resources/Personnel Organization

The interests of each stakeholder will differ, depending on the security aspects of his role and on his position within an organization’s organizational hierarchy. Stakeholder interests may be determined through multiple venues, such as interviews, brainstorming sessions, and meetings. Each stakeholder may require an additional set of customized metrics that provides a view of the security program’s security performance within his area of responsibility.

The total number of metrics should be between five and ten for each individual stakeholder. Fewer metrics per stakeholder are recommended as an organization is maturing its security program. The number of metrics per stakeholder will increase gradually with the maturity of the security program and of the metrics program.

Stakeholders should be involved in each step of security metrics development to ensure organizational acceptance of the concept of measuring security performance. Stakeholder involvement will also ensure that the sense of ownership of the security program security metrics exists at multiple levels of the organization. This will encourage the overall success of the program.

The four measurable aspects of security (business input, efficiency, effectiveness, and implementation) speak to different stakeholders. An executive might be interested in the business and mission impact of security activities. (what is the monetary and public trust cost of the latest security incident? Is there a security-related article about us in a major publication?) Security and program managers may be interested in the effectiveness and efficiency of the security program and its security controls and processes. (could we have prevented the incident, and how fast did we respond to it?) System or network administrators may want to know what went wrong with implementation (have we performed all necessary steps to avoid or minimize the impact of the incident?).

Goals and Objectives Definition

Phase 2 of the metrics development process in Figure 3 is to identify and document the performance goals and objectives that would guide security control implementation.

Applicable documents should be reviewed to identify and extract applicable security performance goals and objectives. Various documents can be used, when appropriate, as sources. The extracted goals and objectives should be validated with the security program stakeholders to ensure stakeholder acceptance and participation in the metrics development process.

Security Policies, Guidance, and Procedures Review

The details of how security controls should be implemented are usually described in organization-specific policies, standards, and procedures (Phase 3 in Figure 3). These define a baseline of security practices. Specifically, they describe security control objectives and techniques that should lead to accomplishing security performance goals and objectives.

These documents should be examined during initial development. They also should be examined in future metrics development, when the initial list of metrics is exhausted and other metrics needs to replace them. The applicable documents should be reviewed to identify prescribed practices, relevant targets of performance, and detailed security controls for system operations and maintenance.

System Security Program Implementation Review

In Phase 4 of the metrics development process illustrated in Figure 3, a review should take place of any existing metrics and data repositories that can be used to derive metrics data. Following the review, applicable information should be extracted and used to identify appropriate implementation evidence that will support metrics development and data collection.

Implementation evidence points to aspects of security controls that would indicate the security performance objective is being met, or at least that actions leading to the accomplishment of the performance objective in the future are being performed. The security requirements, processes, and procedures that have been implemented can be extracted by consulting multiple sources, including documents and interviews and through observation.

If metric data does not have associated implementation evidence identified and documented, it should not be fully trusted. For metrics to be quantified, they need to not only be represented numerically but not be subjective in nature.

Metrics Development and Selection

Phases 5, 6, and 7 in Figure 3 pertain to development of metrics that measure process implementation, effectiveness and efficiency, and mission impact. Implementation evidence, required to prove higher levels of effectiveness, will change from establishing the existence of policies, standards and procedures, to quantifying implementation of these policies, standards and procedures, to quantifying the results from implementation of policies, standards and procedures, and ultimately to identifying impact of implementation on the organization’s mission.

The universe of possible metrics, based on existing policies, standards and procedures, will be quite large. Metrics must be prioritized to ensure that the final set selected for initial implementation has the following qualities:

• The metrics assist improvement of high-priority security control implementation. High priority may be defined by the latest results of risk assessments or an internal organizational goal.
• They use data that can realistically be obtained from existing processes and data repositories.
• They measure processes that already exist and are relatively stable.

Measuring nonexistent or unstable processes will not provide meaningful information about security performance. Therefore, such measurements will not be useful for targeting specific aspects of performance.

A phased approach is required to identify short-, mid-, and long-term metrics. In a phased approach, the implementation time frame depends on a combination of system-level effectiveness, priority of the metrics, data availability, and process stability. Once applicable metrics that contain the qualities described above are identified, they will need to be documented in the implementation plan.

Establishing Performance Targets

After applicable metrics are identified and described, performance targets should be identified in an indicator line of the metric form. Performance targets establish a goal by which success is measured. The degree of success is based on the metric result’s proximity to the stated performance target.

The mechanics of establishing performance targets differ for implementation metrics and the other three types of metrics (effectiveness, efficiency, and impact). For implementation metrics, targets are set to 100 percent completion of specific tasks. Setting performance targets for efficiency, effectiveness, and impact is more complex, because these aspects of the security operation do not assume a specific level of performance.

Identified security management personnel will need to apply qualitative and subjective reasoning to determine appropriate levels of security effectiveness and efficiency, and to use these levels as targets of performance for applicable metrics. All organizations desire effective implementation of security controls, efficient delivery of security services, and minimal impact of security events on its mission. However, the associated measurements will be different for different systems, programs, and controls. An organization will need to establish performance targets for relevant metrics, and should be ready to adjust these targets, based on actual measurements, once they are obtained.

An organization may also decide not to set targets for these metrics until the first measurement is collected so that it can be used as a performance baseline. Once the baseline is obtained and corrective actions identified, appropriate and realistic measurement targets and implementation milestones can be defined. If performance targets cannot be established after obtaining the baseline, management should evaluate whether the measured activities and corresponding metrics are providing the expected value for the an organization security program.

Feedback within Metrics Development Process

The metrics that are ultimately selected for implementation will be useful not only for measuring performance, identifying causes of unsatisfactory measurements, and pinpointing improvement areas. They also will be useful for assisting in continuous policy implementation, affecting security policy changes, and redefining goals and objectives. Once the measurement of security control implementation begins, subsequent measurements can identify performance trends, and ascertain whether the rate of implementation is appropriate.

Once effectiveness and efficiency metrics are implemented, they will assist in understanding whether the security control performance goals, set in the security policies, standards and procedures, are realistic and appropriate. For example, if a security policy defines a specific password configuration, compliance with this policy could be determined by measuring the percent of passwords that are configured according to the policy. Such a measure addresses the level of security control implementation. It is assumed that configuring all passwords according to the policy will significantly reduce, if not eliminate, system compromises through broken passwords.

To measure effectiveness of the existing password policy implementation, the percentage of crackable passwords, by common password breaking tools, could be identified. This measure addresses the effectiveness of the security control as implemented. If a significant percentage of crackable passwords remain after the required password policy has been implemented, the underlying policy may be ineffective in thwarting password compromises. If so, an organization will need to consider strengthening the policy or implementing some other mitigating features. An organization will then need to determine the costs and benefits of keeping the password policy as is, tightening it, or replacing password authentication with other techniques.

Conducting cost-benefit analyses will generate business impact metrics that will address the redefining of system identification and authentication objectives, and appropriately realigning these objectives with the mission of security program.

Metrics Program Implementation

Implementation of security metrics involves using security metrics for monitoring security control performance. It also involves using the results of the monitoring to start performance improvement actions. The implementation process is iterative, and consists of six phases. These phases, when fully executed, will ensure continuous use of security metrics for performance monitoring and improvement of security controls. Figure 7 depicts the security metrics program implementation process.

Phase 1: Prepare for Data Collection

Phase 1 of the process involves activities that are crucial for establishing a comprehensive security metrics program. These include the security metrics identification, definition, development, and selection activities and development of a metrics program implementation plan.

After the metrics have been identified, specific implementation steps should be defined on how to collect, analyze, and report the metrics. These steps should be documented in the Metrics Program Implementation Plan.

The following items may be included in the plan:

• Metrics roles and responsibilities, including responsibilities for data collection, analysis, and reporting
• Audience for the plan
• Process of metrics collection, analysis, and reporting, tailored to the specific organizational structure, processes, policies, and procedures
• Creation or selection of data collection and tracking tools
• Modifications of data collection and tracking tools
• Metrics summary reporting formats

Phase 2: Collect Data and Analyze Results

Phase 2 of the process involves activities essential for ensuring the collected metrics help gain an understanding of security program security and identify appropriate improvements.

This phase includes the following activities:

• Collect metrics data, according to the processes defined in the Metrics Program Implementation Plan.
• Consolidate collected data and store it in a format conducive to data analysis and reporting, for example, in a database or a spreadsheet.
• Conduct gap analysis: compare collected measurements with targets, if defined, and identify gaps between actual and desired performance.
• Identify causes of poor performance.
• Identify areas requiring improvement.

The causes of poor performance can often be identified using the data from multiple metrics. To determine the cause of low compliance, information must be collected on the reasons for the low percentages (e.g., lack of guidance, insufficient expertise, or conflicting priorities). This information can be collected as separate metrics or as implementation evidence. Once this information is collected and compiled, corrective actions can be taken.

The following are examples of causation factors that contribute to poor security control implementation and effectiveness:

• Resources: Insufficient human, monetary, or other resources
• Training: Lack of appropriate training for the personnel installing, administering, maintaining, or using the systems
• Configuration Management Practices: New or upgraded systems that are not configured with required security settings and patches
• Awareness and Commitment: Lack of management awareness or commitment to security
• Policies, Standards and Procedures: Lack of policies, standards and procedures that are required to ensure existence, use, and audit of required security functions
• Architectures: Poor system and security architectures that make systems vulnerable
• Inefficient processes: Inefficient planning processes that influence the metrics

Phase 3: Identify Corrective Actions

Phase 3 of the process involves the development of the roadmap of how to close the implementation gap identified in Phase 2.

This phase includes the following activities:

• Determine range of corrective actions: Based on the results and causation factors, identify corrective actions that could be applied to each performance issue. Corrective actions may include: changing system configurations; training security staff, system administrator staff, or regular users; purchasing security tools; changing system architecture; establishing new processes and procedures; and updating security policies.
• Prioritize corrective actions based on overall risk mitigation goals: There may be several corrective actions, applicable to a single performance issue. However, some may be inappropriate if they are too costly or inconsistent with the magnitude of the problem.
• Select most appropriate corrective actions: Up to three corrective actions from the top of the list of prioritized corrective actions should be selected for conducting a full cost-benefit analysis.

Phase 4: Develop Business Case

Phases 4 and 5 both address the budgeting cycle for obtaining resources required for implementing remediation identified in Phase 3. The results of the prior three phases will be included in the business case as supporting evidence.

The following activities should be performed as a part of business case analysis:

• Document mission and objectives identified during Phase 2 of the metrics development process.
• Determine the cost of maintaining status quo, to use as the baseline for comparing investment alternatives.
• Document gaps between target performance and current measurements, identified during Phase 2.
• Estimate lifecycle cost for each corrective action or investment alternative, identified in Phase 3.
• Perform sensitivity analysis to discern which variables have the greatest effect on cost.
• Characterize benefits that are quantifiable and non-quantifiable returns which are delivered through improved performance, based on the prioritization of corrective actions performed in Phase 3.
• Perform risk analysis to take into account the likelihood of obstacles and programmatic risks of a particular alternative.
• Prepare budget submission by summarizing key aspects of the business case to accurately depict its merits.

In most cases the previous list of activities to create a business case is too time-consuming and overwhelming. It will be up to an organization to determine the requirements and process for this type of business case development. The previous list should be used as a guideline.

Phase 5: Obtain Resources

Phase 5 involves the following activities:

• Responding to budget evaluation inquiries
• Receiving allocated budget
• Prioritizing available resources, assuming that not all requested resources will be allocated
• Assigning resources to perform corrective actions

Phase 6: Apply Corrective Actions

Phase 6 of the process involves implementing corrective actions in technical, management, and operational areas of security controls. After corrective actions are applied, the cycle completes itself and restarts with a subsequent data collection and analysis.

Iterative data collection, analysis, and reporting will track progress of corrective actions, measure improvement, and identify areas for further improvement.

The iterative nature of the cycle ensures that progress is monitored, and that the corrective actions are affecting system security control implementation in the intended way. Frequent performance measurements will ensure that, if corrective actions are not implemented as planned, or if their effect is not as desired, quick course corrections can be made internal to the organization.

Summary

Developing a full security metric program may seem overwhelming. It is important to realize that the development of this type of program starts small and grows in a controlled and disciplined manner. A full security metric program commonly takes two years to fully mature, but that all depends upon the organization’s commitment and available resources.

Appendix

Metric Development Template

© 2024 Pearson Education, Pearson IT Certification. All rights reserved.

800 East 96th Street, Indianapolis, Indiana 46240