A long promising approach comes of age
I won’t revisit the arguments for anomaly detection as a crucial piece of cybersecurity. We’ve seen waves of anomaly detection over the years — and CISA, DARPA, Gartner, and others have explained the value of anomaly detection. As rules-based detections show their age and attackers adopt AI to accelerate their innovation, anomaly detection is more needed than ever.
However — traditional anomaly detection has been caught in a Sisyphean cycle: build multiple bespoke models for each account, tune endlessly, and still face diminishing returns as threats and the operating environment evolve.
In a recent paper, “Expectations Versus Reality: Evaluating Intrusion Detection Systems in Practice” (arXiv:2403.17458), the authors compare traditional machine learning based anomaly detection approaches to a deep neural network (DNN)-based system across several benchmarks: CICIDS2017, UNSW-NB15, and others. The DNN outperformed across all datasets except for one, delivering both higher F1 scores and more consistent performance across different evaluation datasets.
You can see from the paper that the deep neural network (DNN) outperformed all prior generation ML both in terms of accuracy, as measured by F1 score, and adaptability, as shown in having the most consistent performance. The Stratosphere use case where the DNN performed least well is a very specific IoT dataset. It appears that the DNN had not been exposed to this sort of data in its training.
The conclusion is clear: better models are both more accurate AND generalize better.
The Monolithic Past: Fragile, Overfit, and Labor-Intensive
Traditional anomaly detection relies on one or more bespoke models for every environment.
Every model deployed requires:
- Hand-tuned thresholds.
- Manual feature engineering.
- Endless maintenance including retraining and evaluation to combat obsolescence.
Also, these systems suffer from overfitting, locking into patterns specific to the training data but brittle in the face of novel threats. Worse, they’re resource hogs, demanding constant attention from detection engineers. While anomaly detection usage is increasing, it is easy to see why many detection engineers find the
[…]
Content was cut in order to protect the source.Please visit the source for the rest of the article.
Read the original article: