A Brief History of Time

Historically, successes and failures of a security product were evaluated using binary logic ("detected" or "missed") and represented as a detection rate. Such rates are assumed to represent the quality (probability) of successful protection. But the timing of providing protection greatly affects this probability. We would demonstrate with examples that the "detection rate" which does not incorporate the timing element is not a valid metric.
We analyze the factors contributing to the probability of successful protection, present the mathematical approach to calculating this probability and discuss how this can be implemented in practice. We will show that for each attack, the "success of protection" is a function of time. For multiple attacks we will have a set of such functions. We would argue that a simple and meaningful numeric representation of this set of functions would be a probability calculated as an average of integrals of these functions over time.
In this model overall probability depends on the timeframes used to evaluate each attack. To be meaningful, the selection of these timeframes has to take into account users' exposure to the threat. But full knowledge about the exposure is only available after the attack so we have to deal with historical data.