Benchmarking in Security Awareness

Get Started

View Pricing Request Demo

Many people want to know how they are doing compared to their peers. This desire is no exception when it comes to measuring employee security awareness. Organizations would like to know how they are doing when compared to similar organizations. To gauge security posture with peers, the results of one organization are compared to a benchmark or industry average.

A standard testing method for organizations to evaluate security awareness posture is to conduct simulated phishing attacks. Organizations can determine if employees click on links through these simulated attacks, reply to emails, download documents, etc. If employees engage in these kinds of actions with an actual phishing email, the consequences could be severe. In a simulated phishing environment, organizations can track results and provide remedial training to employees as needed. Most organizations will include a "reporting" capability, such as KillPhish, to report suspicious emails. Should a training email be reported, it is considered a plus for the employee. The KillPhish plugin works with the Net Reporter Score, an easy metric for organizations to track security awareness.

The fallacy with benchmarking in security awareness.

There is a significant issue when putting too much reliance on the metric for measuring security awareness. There are many variables involved that are not likely the same across tests. For example, if an organization sends a fairly generic email to everyone vs. a targeted email to a specific department, the results are likely to be significantly different. The failure rate would likely be much higher for the targeted email than the generic email.

Other factors can influence the test results. One common issue is the amount of turnover within a company. If one organization has a higher turnover than the averages of the organizations used in the benchmark, the results will be significantly off. The organization with a high turnover rate will likely have a higher failure rate and lower NRS as its employees have not been through the organization's security awareness training. Consistently adding new employees between simulated phishing tests can skew the metrics of security awareness posture.

What does this mean for benchmarking in security awareness?

When benchmarking in security awareness, excessive focus on benchmarks should be avoided. If people lose their job or receive negative consequences, personnel running the testing may make the test "easier" to avoid the negative consequences. As a result, the training may lose its effectiveness. For example, in a regulated industry, the regulators recommended an organization set specific metrics that would have severe consequences if exceeded. What happened? The security software personnel made sure there were enough "red flags" in the simulated emails that very few employees failed. Other tactics to artificially reduce the failure rate are: to notify people the test is going to occur, sending the same message to many people, sending the message at a specific time, sending the message at an odd time, and sending inappropriate messages (such as invoice to a customer service representative and a support ticket to accounts payable).

What are some industry benchmarks for security awareness training?

Two standard benchmarking metrics are for evaluating security awareness posture are the failure rate and the Net Reporter Score. The failure rate is a simple calculation detailing the percentage of users who failed a phishing test relative to the total number tested. Actions such as entering data on a landing page, opening an attachment, and replying to the simulated phishing email are all causes of a phishing simulation failure. However, if someone does not receive or notice the test email, it would skew the failure rate lower. To solve this problem, the Net Reporter Score was created.

The Net Reporter Score measures the security awareness of an organization by comparing the users who fail to the users who report the suspicious email. The NRS is an easy-to-understand index from -100 to +100. The higher the number, the better the organization's employee security awareness. If a user fails the test, the NRS is lowered. When a phishing campaign is sent to test employees' security awareness, the employees' actions concerning the phishing test are evaluated. Through the KillPhish reporting feature, employees can report when they receive a suspicious email, including test emails. The Net Reporter Score is calculated by subtracting the percentage of people who failed the test from the people who reported the test email. If the employee reports the email, the NRS is increased. If the user does nothing with the email, the results are not impacted. With an NRS, an organization can easily see and monitor the effectiveness of its security awareness training.

Flipping security awareness benchmarking upside down

When most people think about security awareness benchmarking and failure rates from simulated phishing tests, they want to reduce the failure rate or improve the Net Reporter Score. However, as noted above, if the focus is solely on a low failure rate, or high NRS, there can be a tendency to reduce the difficulty to ensure that these metrics are obtained. In a complete reversal, one can actually improve the organization's overall security posture if they try to have high failure rates by making the tests more and more difficult as the security awareness program matures.

How to use benchmarking in security awareness

Benchmarking should be used as general guidance. Overall, the difficulty of the messages should be increased to improve the organization's security awareness continually. However, such an increase in test difficulty could likely increase the failure rate and/or reduce the Net Reporter Score, at least initially.