Security Operations

This research line at TU/e SEC is led by Emmanuele Zambon and Luca Allodi and studies how Security Operations Centers (SOCs) detect and investigate attacks. A SOC combines automated detection, for example network intrusion detection systems (NIDS) such as Suricata, with human analysis of the alerts these systems raise. Both parts have well-known problems. Most alerts turn out to be false positives, entry-level (Tier 1) analysts often work without a shared method and leave the job within a short time, and detection rules are written and maintained with little systematic guidance.

Much research on SOCs proposes automated, often machine-learning based, post-processing of alerts. Our work focuses on two other parts of the pipeline: the analysts who investigate alerts, and how they are trained and supported, and the detection rules that generate the alerts. The research is carried out in close collaboration with commercial SOCs, including the Eindhoven Security Hub SOC, which give access to real alert data, deployed rulesets, and practising analysts and rule engineers as study participants. Most studies combine data from operations with controlled experiments involving analysts or students.

Analyst support

A structured process for Tier 1 analysis

Tier 1 analysts decide which alerts are escalated to more senior analysts. In many SOCs this decision relies mainly on tacit knowledge, so the information collected for an alert, and the quality of the decision, varies between analysts. Playbooks exist, but they are usually tied to specific use cases and are hard to keep up to date.

In [1] we derived, together with Tier 2 and Tier 3 analysts of the SOC, a generic threat analysis process for network alerts. The process has 13 information-gathering steps grouped in four stages: relevance indicators (for example, how specific and how old the triggering signature is, and whether the target is in the customer’s scope), additional alerts, contextual information about the internal host, and evidence of the attack and its success. The process does not prescribe decisions or thresholds. It tells the analyst which evidence to collect before deciding whether an alert is worth escalating.

We evaluated the process in an experiment with ten newly hired Tier 1 analysts of the SOC, who each classified 80 alerts sampled from a real customer network, with injected attacks added to the data. Analysts who followed the process were about 2.5 times more likely to classify an alert correctly than analysts who received the SOC’s standard training. The gain was concentrated on alerts related to actual attacks, which are the ones that matter most for escalation.

Training new analysts

Because Tier 1 turnover is high, SOCs need to train new analysts quickly. Cyber ranges, isolated environments that replay realistic attacks, are a common tool for this. Their design usually puts most effort into technical realism, with less attention to how trainees learn. In [2], together with the University of Regensburg, we kept the technical setup of an existing cyber range exercise used by a commercial SOC (the investigation of a Log4j attack in a SIEM) and replaced its learning management system with T1GER. T1GER applies instructional design principles: automated feedback that explains why an answer is right or wrong, hints that provide procedural guidance when a trainee is stuck, and a pseudonymous scoreboard.

In a randomised controlled trial with 144 students from cybersecurity courses at two universities, trainees using T1GER reported higher motivation and lower cognitive load, were more likely to complete the exercise within the 90 minutes available, and needed less time to do so (65 versus 73 minutes on average). Knowledge outcomes were the same in both groups. Since the redesign reduces the need for a dedicated trainer, it also makes the exercise easier to repeat for each new group of hires. T1GER is available as open source.

Rule engineering

What makes a rule noisy

Previous studies of SOC detection looked at rulesets as a whole. In [3] we analysed individual rules: a ruleset with 290 thousand rule revisions and the 30 million alerts they produced over 11 months at two customers of a commercial SOC, linked to the SOC’s incident reports. About 80% of the rules that fired raised less than one alert per day, while a small fraction caused most of the workload. Over 75% of the incidents were detected by rules matching atomic indicators of compromise, such as malicious domain names in DNS requests. Recently introduced rules and legacy rules that had not been updated for years both contributed to detection.

From the analysis of high-noise rules and interviews with the SOC’s rule designers, we formulated six design principles. Four aim at specificity: detect a characteristic that follows directly from the malicious behaviour rather than a proxy for it, alert on successful malicious actions where this can be observed, throttle alerts, and use exceptions for known benign cases. Two aim at coverage: generalise the matched characteristic, and generalise its position in the traffic. For example, a rule that alerts on any request to the ColdFusion administrator interface can be paired with a second rule that only alerts when the server responds with success:

alert http any any -> $HTTP_SERVERS any (msg:"ColdFusion administrator access";
  flow:established,to_server; http.uri; content:"/CFIDE/administrator"; nocase;
  flowbits:set,coldfusion_admin_access; flowbits:noalert;)

alert http $HTTP_SERVERS any -> any any (msg:"ColdFusion successful administrator access";
  flow:established,to_client; flowbits:isset,coldfusion_admin_access;
  http.stat_code; content:"200";)

Regression on the rules in our data showed that using proxies, not distinguishing successful from unsuccessful actions, and not throttling alerts all significantly increase unnecessary analyst workload. Generalising the matched characteristic also increased workload, so coverage and specificity have to be balanced during rule design. A tool built on the principles indicated that most rules in two widely used open and commercial rulesets probably rely on proxies and do not use exceptions or throttling. The tool is available as open source under the name suricata-check.

How rules are engineered

Since most rules are rarely updated after release, the way a rule is first written largely determines its quality, yet this process had not been studied. In [4] we built SuriCap, a platform on which participants write Suricata rules for attack scenarios, test them against visible test traffic and receive feedback, while hidden tests check whether the rules generalise to variants of the same attack. In capture-the-flag style workshops, 60 participants, ranging from students to professional rule engineers of a SOC, submitted 3,146 valid rules for four scenarios.

Prior experience had no significant effect on rule quality or on how efficiently participants arrived at a working rule. Generalisation was the main difficulty: once rules passed the visible tests, further changes rarely improved their performance on the hidden tests. The submission histories show three phases: planning the rule, implementing it and refining it. Clustering them revealed a common strategy, shared with the professional engineers, in which engineers first fix where and in which direction a characteristic can be observed, and then implement the matching of the characteristic itself. Most updates fine-tuned an existing rule rather than trying an alternative design. For SOCs, these results suggest investing in test data and feedback loops between analysts and rule engineers as a way to improve rule quality.

Outlook

Ongoing work connects the two lines, for example by using rule design principles to improve the alerts analysts receive, and by extending the experimental platforms to other detection languages and to automated support for rule engineering. Results from our cyber threat intelligence research inform which threats detection should focus on.

References

1. L. Kersten, T. Mulders, E. Zambon, C. Snijders, and L. Allodi. ‘Give Me Structure’: Synthesis and Evaluation of a (Network) Threat Analysis Process Supporting Tier 1 Investigations in a Security Operation Center. In Nineteenth Symposium on Usable Privacy and Security (SOUPS), pp. 97–111, 2023. Paper
2. M. Glas, L. Kersten, T. Mulders, G. Pernul, and L. Allodi. T1GER: An Instructional Re-Design of a Cyber Range Exercise in a Commercial Security Operations Center. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ‘26), 2026. doi:10.1145/3772318.3791226
3. K. T. W. Teuwen, T. Mulders, E. Zambon, and L. Allodi. Ruling the Unruly: Designing Effective, Low-Noise Network Intrusion Detection Rules for Security Operations Centers. In ACM Asia Conference on Computer and Communications Security (ASIA CCS ‘25), pp. 1428–1441, 2025. doi:10.1145/3708821.3710823, code
4. K. T. W. Teuwen, E. Zambon, and L. Allodi. How It’s Made: Uncovering Detection Engineering Processes for Network Intrusion Detection Rules. In 11th Workshop on Traffic Measurements for Cybersecurity (WTMC ‘26), 2026. doi:10.1145/3846375.3849098, arXiv:2609.25901