Everything in security boils down to data. Often, you are protecting private, sensitive or valuable data. You use data about data (Metadata). You monitor data about how people use your data (Security Events). You gather data on data in order to focus your attention (Security Alerts). You need data about what bad guys are doing to acquire your data (Threat Intelligence). You even need data about the data that measures your data (Telemetry).  All this data is overwhelming!  You do not want too much data.  You do not want too little data. You want data that is juuuuuuust right. But that’s another story for another day…

This is a story about when you have the right amount of Security Event data but the data is not aligned with the truth or the whole picture. This is the story of Bad Data in Third Party Cybersecurity Monitoring (TPCM). (TPCM is one type of Continuous Monitoring, a best practice for real-time understanding into risks to your organization.)

Data itself can have a strong personality like the wolf who confronts Little Red Riding Hood. Data can appear to be on-target, relevant, reliable and easy to change…or it could be the opposite:

• Data sometimes marauds as something other than itself.  A wolf dressed as grandmother is an example of a Data Accuracy.
• Data should not always be trusted.  Little Red Riding Hood should have analyzed those big, hairy hands and the deep voice using both Prioritization Relevance and Intelligence Reliability.
• In the end of both the TPCM tale and the fairytale, resolving issues around data hinges on swift problem-solving.  Little Red Riding Hood is rescued by Correction Agility – the axe-wielding woodsman who slashes open the wolf.

There are three characters in this story:

• The Vendors – The organizations being monitored
• The Customer – You, who are requesting the monitoring of the Vendor
• The Cyber Monitoring Firm – The organization doing the monitoring of the Vendors for the Customer

Cyber Monitoring Firms are known as Security Ratings Services for the scores they assign to a vendor’s security posture. There are other players in this space that have focus more on your security rather than the security of your vendors. They are referred to as Cyber Threat Intelligence services – for the sake of simplicity, we will call them all Monitoring Firms.

Monitoring Firms gather information around a vendor’s security practices. Monitoring Firms often provide a score or rating, with letters or numbers, much like a report card or a credit score. Monitoring Firms highlight good and bad indicators in their rating systems.  Monitoring firms can notify you when indicators change positively or negatively around how safely Vendor’s access and treat your data.

A Monitoring Firm’s ability to guide vendor selection depends on how the Monitoring Firm interprets and presents a Vendor’s data to you.  The following infographic and text detail the four most pervasive issues in bad data in TPCM:

Accuracy – A security event’s alignment with reality

• Assignment Accuracy – Systems assigned to correct Vendor associated with Customer
• False Positive – Alerts triggered when inappropriate
• False Negative – Alerts not triggered when appropriate

Relevance – Alignment of alerts with customer needs and perspective on risk

• Prioritization Relevance – Prioritization of alerts customized to customer needs
• Prioritization Customization – Degree to which prioritization can be changed for customer

Intelligence Reliability – Trustworthiness of intelligence sources used to create ratings, events and alerts

• Intelligence Sourcing – Percentage of data (Alerts and Organization Assignment) sourced directly from external data feeds vs. sourced/enhanced internally
• Intelligence Modularity – Ability for data to be overridden if it needs to be modified
• Intelligence Standardization – Data communicated in external standardized language/references or proprietary language

Correction Agility – A Monitoring Firm’s ability to nimbly correct issues

• Correction Speed – Rate at which data issue can be corrected, starting from the moment issue is identified
• Correction Workflow – Ease at which a monitoring firm can correct issues
  • How are correction submissions gathered?
  • How is the monitored organization notified?
  • Are clients notified about data under review?
  • How is status of the review communicated?

We hope this introduction to these measures of data quality by way of fairy tale will help you better understand all that goes into vendor security ratings and events. Now you know the characters and the plot – but this is just chapter one! In the weeks ahead, we will dig deeper into each of these areas to better understand the current state of practice and opportunities for improvement across the Third Party Cyber Monitoring field.