From meh to mega-metrics: How to measure the impact of your security awareness training

You move into a new house—congratulations! It’s beautiful. Period features. Original wooden floors. One of those shiny overhead showers. Views for days.

One downside: The basement is really damp. 

No worries—you decide to buy a dehumidifier.

You spend days comparing reviews and specs. You find a top-rated model that can more than handle your home. Sorted.

You read the installation instructions. You plug it in, turn it on, and walk away.

You don’t plan on checking the basement again.

Why should you? The dehumidifier’s purpose is to do the thing you need it to do, where you need it to do it.

Weeks later, as mold creeps up the walls of your ground floor, you realize something: 

That humidifier hasn’t been doing the thing you thought it was doing. Not at all. And now your drapes are green and fuzzy, and your lounge smells like an artisan camembert.

The link between a smelly French cheese and cybersecurity is this: 

You don’t buy a dehumidifier because you want to own a dehumidifier. You buy one because you don’t want a damp room. 

Likewise, you don’t implement a security awareness training program for its own sake. You do it to manage the human risk element in your organization.

So . . . you need to check if it’s actually doing it. 

Not least because—unlike mold—a cyber attack is sudden, instantly destructive, and you can’t see it coming.

How do we know something is doing what we want it to do?

For a dehumidifier it’s easy enough to tell by looking at two things: Is it gathering moisture? Is the space less damp? 

Both of those things are metrics.

But how about cybersecurity awareness training? How do we know that’s working? What are the best security awareness metrics

Listen, to handle ever-evolving cyber threats, security teams need to reduce human cybersecurity risk. And to reduce human cyber security risk, security teams have to measure what matters.

NGL: Measuring is difficult . . . and some measuring is more difficult than other measuring

Good news: The most popular type of security awareness training is also one of the easier ones to measure. 

Not only has the internet allowed us to see the same woman dancing the same dance in a different brightly colored outfit every day . . . it’s made it possible for a security team to see who is doing what, where they’re doing it, and when they’re doing it.

We’re talking about online security awareness training. And it’s by far the most popular form of security awareness training today.

But when training isn’t online, things become trickier to analyze.

Whatever the format, measuring security awareness training effectiveness isn’t an easy nut to crack. Even the most seasoned security leader has often wondered:

We’ve got the answers—as well an alternative approach, which we think could hold the key.

How do organizations measure the effectiveness of security awareness training?

Admit it. You’ve wondered how everybody else is doing it.

It’s a question that has been on the minds of experts for quite some time. As far back as 2012, the Educause Center for Analysis and Research was already looking into it.

In 2013, the Center published the results of a survey that asked 95 universities how they measured the success of their security awareness program. The results were quite interesting, and they reported using the following security awareness metrics:

The fact that the survey took place in the olden times of 2012 makes us even more impressed with the participants. Especially since many were measuring impact in more than one way. 

But what was impressive in 2012 isn’t so impressive now. Just ask Vine and cronuts.

Today’s organizations need to be on it with their metrics. On. It.

Security awareness metrics: What to measure . . . and how

Let’s run through some common metrics, the challenges of measuring security awareness training, and what a security awareness metrics matrix looks like and what it does.

Common cybersecurity awareness metrics to track

We’ll go into more depth later, but here’s a little appetizer:

Participation rates

This is a common metric used to measure the number of people who complete the security awareness training. Why? Tracking participation rates can help you identify whether your people are willing to participate in the training. If engagement is dismal, you can make improvements to increase participation rates.

Completion rates

This is the percentage of people who complete the entire security awareness training program. Measuring completion rates is a no-brainer for making sure people have access to and complete the training program in its entirety.

Quiz scores

These will reveal a picture of people’s knowledge retention. By tracking quiz scores, you can determine whether people are retaining the information delivered through your security awareness training program.

Phishing simulation results

This is a metric used to measure people’s susceptibility to phishing attacks—and it can also tell you something about how vulnerable someone is to social engineering. By sending phishing simulations to people, you can determine things like the percentage who fall for the phishing attack and areas of improvement. Then you can tweak your training content as needed.

Pro tip: While we’re on social engineering, many people don’t realize how skilled threat actors are at using social media to craft convincing phishing emails that mimic real communications from trusted sources. Therefore, it’s crucial to include social media awareness in your security awareness program. It’ll stop a few security threats in their tracks.

What are the main challenges of measuring security awareness training?

Measuring the effectiveness of security awareness training can be tough. And that’s because of several different things. Here are the most common headaches:

Metrics that matter

It can be challenging to determine which metrics to track. It’s essential to focus on the metrics that matter most to your organization and align with your security goals and objectives.

People persuasion

People may not be willing to participate in the training, making it difficult to get accurate participation metrics.

Changing behaviors

Awareness does not equal behavior change. People may not take the training seriously or may not retain the information, making it challenging to measure engagement and behavior change metrics accurately.

Security posture impact

Measuring the impact of the security awareness program on the overall security posture of the organization can be challenging since many other factors can impact an organization’s vulnerability to the whole security issue spectrum.

Security awareness metrics matrix

So, how to overcome these challenges?

A security awareness metrics matrix can help.

This matrix can help you define the metrics you need to track, set specific goals, and measure your progress towards those goals. Your matrix should include:

1. Key performance indicators (KPIs): Metrics that measure the performance of your security awareness program.
2. Targets: Specific goals that you want to achieve for each KPI.
3. Metrics: Specific data points that you’ll track to measure progress towards your targets.
4. Frequency: How often you’ll collect data for each metric.
5. Owners: The individuals or teams responsible for tracking and reporting on each metric.

Your security awareness metrics matrix can help you identify areas of improvement and track your progress towards your goals. With this information, you can make data-driven decisions and ensure the long-term success of your security awareness program.

(We got to the end of the bit about a matrix without referencing Keanu Reeves in a trenchcoat. Please clap.)

The right way measure the effectiveness of security awareness training

What if we told you . . . that creating a matrix and monitoring metrics (phew, say that after a few beers) aren’t enough?

Hold up—don’t pack it in just yet.

It takes more. But we’ve got a plan.

Step 1: Strategy

A coherent and well-planned cybersecurity strategy. It’s a must for measuring the effectiveness of cybersecurity training. 

The reason’s simple. Without a strategy, any security awareness program runs the risk of becoming a fleeting effort.

So . . . before thinking about measurement, think about what you want to achieve and how you want to achieve it.

Let’s say you want to reduce the likelihood of people leaving computer screens unlocked when away from their desk by 50% over the next six months. 

How might you do that? Posters? Prizes? Sticking googly eyes to monitors to influence the subconscious?

Step 2: Questions

Time for some soul-searching.

Because you, as your organization’s cybersecurity guru, need to reflect on what’s really important to find out. And we’re talking in qualitative terms here. Don’t get obsessed with one small part of the puzzle.

Keep an eye on a few things simultaneously: Security awareness, security behaviors, and security culture. AKA the ABC of information security.

What does this mean for you? You need to factor in:

• What people know and understand about how to stay safe online (information security awareness).
• How people really behave when presented with attacks (behavior).
• What people think about, how much they care about, and how confident they are about cybersecurity (culture).

Let’s take it back to the locking screens issue. To improve the incidence of screen-locking, what do we need to find out?

• On the awareness front, it’s natural that any security leader wants to know how much people know about the cybersecurity risk landscape and best practices. 
• But when it comes to behaviors, it’s about how likely it is for an unattended screen to be left unlocked.
• And when it comes to culture, you’ll probably want to find out why people are behaving in the way they are.

Step 3: Metrics

Measuring awareness

If you’ve made it this far then you’re closer than you might think to closing in on those all-important helpful metrics to monitor. 

Take the matter of what people know and understand about how to stay safe, for example.

People’s understanding of security can be monitored through online security awareness training performance. 

So long as you have access to a cyber awareness platform with analytical capabilities, it becomes easy to see how much people know about security best practices. 

Online quiz results can reveal whether people know of the risks of leaving unattended monitors unlocked. And you can dig into figures at an organizational level, a departmental level, and an individual level.

Measuring behavior

Measuring behavior, meanwhile, is usually best achieved through simulated attacks, such as phishing email campaigns.

Simulated attacks test people’s security behaviors. And that means that by monitoring how people respond to simulations, you can use the metric of security behavior.

A word of warning though: A simulated attack may not be appropriate when attempting to measure the chances of an unattended computer screen being unlocked at any given time. 

But a coherent strategy makes finding alternatives easy. Spot checks, for example, should give you the information you need. 

How about incentives? It’s an option, and it’ll work in some situations—you’re the best judge of whether it could benefit your organization. One example is using yellow cards for unattended unlocked screens, versus chocolate bars for unattended screens that have been locked!

Indications of behavior change can also be measured in other ways. We know triggers and motivations are key to behavior change. So how can you measure these in your organization?

Measuring culture

We won’t mince our words. Measuring culture is the hardest-to-measure of the ABCs.

But it’s not impossible. 

Anonymous surveys, for example, can give you an idea of why people take risky actions like leaving their screens unlocked. And the answers can be revealing.

It can tell us about Shaun.

You see, Shaun always leaves his screen unlocked when he walks the couple of steps to the office printer. All his colleagues do the same. It’s an unwritten rule. Because the risks are so low.

And sure, in theory, it’s low-risk. 

But should the printer unexpectedly need more paper, risks increase.

And how about if a visitor passing through needs some help operating the water dispenser?

What if that visitor is distracting Shaun on purpose so their buddy can access Shaun’s computer?

If you can find out about a culture, you know what needs to change in that culture..

Qualitative insights from surveys can help you change behaviors and reduce risks.

But . . . the key is finding an overall quantitative cultural metric. That’s where the rubber hits the road and solid improvements can be made.

Step 4: Timing

When measurements are taken is also particularly important.

In an ideal world, you’d capture the situation at day zero. 

But let’s face it, how many times have you got partway through repainting a room in your home only to realize you didn’t take any before photos? 

So if you’re midway through a security awareness campaign and didn’t measure anything from the beginning, don’t sweat it.

Do the next best thing. Start now. Record measurements at regular intervals as the campaign rolls on. Ideally do this monthly. If you can do it even more frequently, even better.

Why? Regular measurement:

• Fuels continuous improvement
• Helps identify incidents and where people may need further support
• Ultimately prevents many a security incident and data breach.

There’s no doubt. Regular measurements bolster information security.

Wait—what about the number of security incidents?

The Educause survey results show it was the most popular measurement. 

Sure, it’s useful. But is it as useful as its popularity suggests?

Does it really give a clear idea of the effectiveness of security awareness training?

The thing is, the metric fails to take into account the severity of attempted attacks, the resulting statistic can be misleading. 

What do we mean? Well, say the number and severity of attacks increases while training is delivered. It wouldn’t be surprising if the number of incidents increased. And this wouldn’t necessarily mean that the cybersecurity training is ineffective.

Here’s the next step in your metrics mastery. . .

Measuring the effectiveness of security awareness training can be challenging . . . but it doesn’t have to be. 

Like that dehumidifier in the basement, your training program exists to solve a problem.

By tracking the right metrics, engaging people, and measuring behavior change, organizations enhance their security posture.

To make it easier to become a master of measurements we’ve put together a comprehensive guide on how to do just that. Our free ebook on how to measure behavior offers practical tips and insights to help you measure the effectiveness of your organization’s security awareness training.

Effective training is a sizable investment in your organization’s security posture. 

And when you know that investment is working, you know it’s paying dividends in the form of reduced risk, increased productivity, and peace of mind.

And you know your people are making better decisions, following best practices, and avoiding common pitfalls. 

And above all, you know your organization has a formidable human line of defense against cybersecurity threats.