Security Metrics Development and Implementation Based on NIST Directives

Metrics are tools that should be used to aid in decision making, and improve performance and accountability through collection, analysis, and reporting of relevant performance-related data.

Security metrics are based on security performance goals and objectives. Security performance goals state the desired results of implementation of a security program. Security performance objectives, in turn, enable the accomplishment of goals. They do this by identifying practices defined by policies, standards and procedures that direct consistent implementation of data protection controls across the organization.

The policies, standards, and procedures describe the controls (technology, process, administrative) that should be in place, and metrics provide insight into the implementation, efficiency, effectiveness, and business impact of these controls. Before beginning the process of developing a security metric program, an organization first needs to get the proper policies, standards, and procedures developed and in place—otherwise there is nothing to use as benchmarks.

Security metrics monitor the accomplishment of the goals and objectives outlined in the stated documents. They accomplish this by quantifying the level of implementation of the security controls and the effectiveness and efficiency of the controls, analyzing the adequacy of security activities, and identifying possible improvement actions.

The following matters must be considered during development and implementation of a security metrics program:

• Metrics must yield quantifiable information (percentages, averages, and numbers).
• The data that supports the metrics needs to be readily obtainable.
• Only repeatable processes should be considered for measurement.
• Metrics must be useful for tracking performance and directing resources.

The metrics development process, as described below, ensures that metrics are developed with the purpose of identifying causes of poor performance, and that they therefore point to appropriate corrective actions.

Types

An organization should develop and collect metrics of three types:

• Implementation metrics to measure implementation of security controls
• Effectiveness/efficiency metrics to measure the results of security controls
• Impact metrics to measure the impact on business or mission of security events

The types of metrics that can realistically be obtained and are useful for performance improvement depend on the maturity of the organization’s security program. Although different types of metrics can be used simultaneously, the primary focus of security metrics shifts as the implementation of security controls matures.

It cannot be emphasized enough that great diligence must be taken when developing initial metrics. Capturing the wrong type of data ends up in a waste of time and resources. Capturing partial data shows only part of the story. And capturing data that does not have supportive evidence provides a false sense of security.