ML-Based Security Analytics in NetFlow Analyzer: Detecting Advanced Network Threats Through Behavioral Analysis

This article explores how machine learning (ML) and network flow telemetry can be used to establish behavioral baselines, detect anomalous asset activity, and map suspicious network behavior to the MITRE ATT&CK framework. We'll also walk through a realistic security investigation showing how reconnaissance, unusual internal communication, and low-and-slow data exfiltration can be identified through changes in an endpoint's normal traffic behavior.

Modern cyberattacks don't always trigger immediate firewall or IDS/IPS alerts. Once an attacker compromises a legitimate endpoint, reconnaissance, lateral movement, and data exfiltration can occur quietly through network traffic that may appear legitimate when examined in isolation.

This raises an important question for network and security teams:

How do you identify potentially malicious activity when the individual network connections don't immediately look malicious?

A modern NetFlow Analyzer can help answer this by examining NetFlow, IPFIX, sFlow, and other flow telemetry already generated by network infrastructure. When combined with ML-driven behavioral baselining, anomaly detection, risk scoring, and MITRE ATT&CK-aligned detections, this telemetry can reveal deviations from an asset's normal network behavior that traditional signature- or threshold-based monitoring might overlook.

Key Topics Covered

• Why Advanced Threats Can Be Difficult to See
• From NetFlow Telemetry to Security Analytics
• Understanding ML-Driven Behavioral Baselining
• Moving From Anomaly to Attack Context With MITRE ATT&CK
• How Security Analytics Complements Traditional Network Defenses
• Practical Investigation: Detecting a Silent Data Exfiltration Attack
• Why Flow Analytics Is Particularly Valuable for Security Teams
• Summary

Related Articles:

• Complete Guide to Netflow: How Netflow & its Components Work. Netflow Monitoring Tools
• Netflow: Monitor Bandwidth & Network Utilization. Detect LAN, WAN, Wi-Fi Bottlenecks, Unusual Traffic Patterns, Problems and more
• NetFlow Analyzer: Free Download, Step-by-Step Installation, Configuration & Optimization Windows - Linux
• Netflow vs SNMP. Two Different Approaches to Network Monitoring

Why Advanced Threats Can Be Difficult to See

Consider a relatively common attack sequence. An attacker may gain an initial foothold inside the network through several methods, including:

1. Phishing / Credential Compromise

A user unknowingly exposes credentials or executes a malicious payload through a convincing phishing attack.

2. Internet-Facing Application Exploit

An attacker exploits a vulnerable external service or application to gain an initial foothold inside the network.

In either case, the attacker now has access to an endpoint inside the organization.

Rather than immediately launching an obvious attack and increasing the likelihood of detection, the attacker can begin quietly learning about the environment.

The compromised host might:

• Contact internal IP addresses it has never previously communicated with.
• Probe multiple TCP or UDP ports.
• Query internal infrastructure and services.
• Establish connections to file servers or other high-value systems.
• Communicate outside normal operating hours.
• Begin transferring information to a new external destination.

None of these activities, however, necessarily proves that an attack is occurring.

A workstation contacting a server is normal. A device establishing an outbound HTTPS session is normal. An administrator connecting to multiple internal systems may also be perfectly legitimate.

The problem appears when these behaviors are examined in context.

Consider a workstation that normally communicates with ten predictable internal services. If it suddenly begins contacting dozens of systems, the change is worth examining. If the same activity occurs at 2:00 a.m., it becomes more unusual. If the workstation then begins sending data to an external destination it has never contacted before, the combination of these behavioral changes becomes considerably more significant.

This is where behavioral network analytics provides a different perspective from traditional signature-based detection: individual activities may appear legitimate, while the sequence and context surrounding them can reveal potentially malicious behavior.

From NetFlow Telemetry to Security Analytics

NetFlow Analyzer's Security Analytics operates on flow telemetry including NetFlow, IPFIX, sFlow, and other supported flow technologies generated by network infrastructure.

Flow records provide metadata describing network conversations rather than the complete packet payload.

Depending on the flow technology and exporting device, this can provide information such as:

• Source and destination IP addresses
• Source and destination ports
• Transport protocol
• Traffic volume
• Communication direction
• Flow duration
• Conversation frequency
• Applications or other traffic classifications available from the exporter

For security analysis, this metadata is extremely valuable because attacks still create network behavior.

An attacker can encrypt an HTTPS connection, for example, but encryption does not make the connection itself invisible. Flow telemetry can still reveal that a host communicated with a destination, when the communication occurred, and how much traffic was exchanged.

Security Analytics builds on this visibility by examining network behavior at the asset level.

One important design consideration described by ManageEngine is that assets are not tracked solely by IP address. IP addresses can change because of DHCP leases, VPN connectivity, or device reassignment. Security Analytics can use hostname and MAC address information obtained through DHCP to maintain persistent context around an asset.

This allows historical behavior and associated risk to follow the device rather than treating every IP address change as an entirely new entity.

Understanding ML-Driven Behavioral Baselining

Behavioral baselining is one of the most important concepts behind Security Analytics.

Traditional monitoring commonly depends on predefined thresholds.

For example: Generate an alert when outbound traffic from a host exceeds 1GB.

This can be useful, but it has an obvious limitation. The threshold has no understanding of what is normal for the device generating the traffic.

One gigabyte of outbound traffic might be completely ordinary for a backup server but highly unusual for a receptionist's workstation.

Behavioral analytics approaches the problem differently.

Instead of asking only: Did this device cross a predefined threshold?

it can ask: Is this behavior unusual for this particular device?

When an asset is monitored, ML models analyze characteristics of its normal network activity. According to the Security Analytics functionality described by ManageEngine, this includes factors such as:

• When the device is normally active
• Which internal systems it typically communicates with
• Which external systems it communicates with
• Communication frequency
• Typical traffic volume
• Protocols and applications normally used
• Normal data-transfer behavior

Together, these observations establish a behavioral profile for the asset.

Why a Dynamic Baseline Matters

Enterprise networks are not static.

Applications are upgraded. Employees change roles. New cloud services are introduced. Workloads move between systems. Remote-working patterns change. Servers may experience different loads at different times of the month.

A behavioral baseline therefore cannot simply be established once and treated as permanent.

NetFlow Analyzer uses adaptive behavioral baselining, allowing its models and thresholds to recalibrate as normal network behavior evolves.

This is important for reducing one of the largest operational problems facing security teams: false positives and alert fatigue.

A static rule might repeatedly flag legitimate activity simply because it exceeds a predefined value. An adaptive baseline attempts to establish whether the activity represents a genuine deviation from the asset's historical behavior.

Anomaly Does Not Automatically Mean Attack

There is an important distinction worth making.

Unusual behavior is not necessarily malicious behavior.

A user uploading a large file to a new cloud service may represent a perfectly legitimate business activity. A server administrator performing emergency maintenance at 3:00 a.m. might generate highly unusual network behavior without any security incident taking place.

Behavioral analytics therefore provides an investigative signal, not automatic proof of compromise.

Its value becomes much greater when several behavioral deviations appear together.

Consider the following example:

A workstation:

1. Becomes active at an unusual time.
2. Contacts several internal systems it does not normally access.
3. Establishes communication with a previously unseen external host.
4. Transfers significantly more outbound data than its normal baseline.

Any one of these observations could have an innocent explanation.

Taken together, however, they describe a pattern that deserves immediate investigation.

This ability to correlate deviations around the behavior of an asset is where ML-assisted network analytics can provide considerable value to SOC and security teams.

 

Moving From Anomaly to Attack Context With MITRE ATT&CK

Finding unusual network behavior solves only part of the security problem.

An analyst still needs to understand what that behavior might represent.

This is where the MITRE ATT&CK framework becomes useful.

MITRE ATT&CK organizes observed adversary behavior into tactics and techniques. Instead of simply reporting that "unusual traffic occurred," ATT&CK-oriented analysis can help place the observed behavior within the broader context of how attackers operate.

Security Analytics maps detections to MITRE ATT&CK, providing analysts with additional context when investigating suspicious activity.

Putting Network Behavior Into an Attack Sequence

Imagine again that our compromised workstation begins interacting with the network.

Initially, the attacker probes the internal environment looking for systems and services.

From an attack-analysis perspective, this behavior may be associated with Discovery activities.

The attacker subsequently begins communicating with other systems in an attempt to extend access through the environment. The investigation may now reveal behavior associated with Lateral Movement.

Later, data is gathered and transmitted outside the organization. The analyst is now dealing with behavior associated with the Exfiltration stage of an attack.

Conceptually, our investigation of begins to look like this:

This context is significantly more useful than treating every unusual flow as an unrelated alert.

It allows the analyst to ask a more important question:

Are we looking at an isolated anomaly, or are we observing different stages of an active attack?

MITRE ATT&CK mapping can therefore help bridge the gap between network anomaly detection and incident investigation.

 

How Security Analytics Complements Traditional Network Defenses

Security Analytics should not be viewed as a replacement for firewalls, IDS/IPS platforms, endpoint detection, or threat intelligence.

These technologies answer different security questions.

A firewall might ask:

Is this connection permitted by policy?

An IDS/IPS may ask:

Does this traffic match a known malicious pattern?

Threat intelligence might ask:

Is this destination already known to be malicious?

Behavioral analytics introduces another question:

Is this asset behaving differently from the way it normally behaves?

That distinction becomes particularly important when dealing with previously unknown infrastructure, compromised legitimate credentials, insider threats, and attack techniques that do not produce an obvious known signature.

Core Security Analytics Capabilities

NetFlow Analyzer brings several capabilities together to support this approach.

Detect Unknown Threats Without Depending Exclusively on Signatures

Signature-based controls are extremely useful when the malicious activity is already understood.

Behavioral analytics provides another detection mechanism by looking for deviations from established asset behavior. This can surface suspicious activity even when a known malware signature or threat-intelligence indicator is unavailable.

Reduce Alert Noise With Adaptive Baselines

Dynamic baselines allow events to be evaluated relative to each asset's historical behavior.

This is fundamentally different from applying identical static thresholds to every device regardless of its role or normal traffic profile.

Add ATT&CK Context to Security Events

MITRE ATT&CK mappings help analysts understand how detected activity may correspond to known adversary tactics and techniques, giving the event more meaning during an investigation.

Consolidate Investigation Details

Security Analytics provides event information including offenders, victims, rule violations, severity, MITRE mappings, and associated traffic details within a consolidated view.

Rather than beginning an investigation with only an isolated alert, analysts can examine the surrounding network context and determine whether further response is required.

Practical Investigation: Detecting a Silent Data Exfiltration Attack

Let's put these concepts together in a more realistic scenario.

Assume an employee workstation has been compromised.

The attacker does not immediately attempt to transfer gigabytes of information outside the network. Doing so could trigger bandwidth thresholds or other monitoring controls.

Instead, the attacker takes a low-and-slow approach.

Stage 1 – Establishing Normal Behavior

Before the compromise, Security Analytics has already observed the workstation's normal network behavior.

The endpoint typically:

• Operates during normal business hours.
• Communicates with a predictable group of internal servers.
• Uses a relatively consistent set of applications and protocols.
• Exchanges relatively modest quantities of outbound data.
• Communicates with a recurring set of external services.

None of these characteristics is inherently a security rule.

Together, however, they form the behavioral baseline against which future activity can be evaluated.

Stage 2 – The Attacker Begins Reconnaissance

Following the compromise, the workstation begins communicating with internal systems it rarely or never previously contacted.

The attacker is trying to understand the network.

A few connections might not be particularly significant. But communication with many unfamiliar internal hosts or services represents a change from the workstation's established behavior.

The first anomaly has appeared.

Stage 3 – The Behavior Changes Further

The attacker identifies useful systems and continues operating from the compromised endpoint.

Additional deviations begin appearing.

The workstation may communicate at unusual times, interact with systems outside its normal peer group, or exhibit traffic patterns inconsistent with its historical profile.

Again, none of these events individually proves compromise.

But the number of deviations is increasing.

For an analyst, the investigation has moved beyond:

"This workstation generated unusual traffic."

The more useful observation is now:

"This workstation is demonstrating several behavioral changes consistent with activity that warrants investigation."

Stage 4 – Slow Data Exfiltration Begins

After identifying and collecting useful information, the attacker begins transmitting data to an external host.

Importantly, the attacker deliberately keeps the transfer volume low.

There is no massive bandwidth spike.

The external IP address has not necessarily appeared on a threat-intelligence feed.

The traffic may use an otherwise legitimate protocol.

There may be no malware signature for the network security controls to identify.

If the connection is examined individually, it may look unremarkable.

But Security Analytics has historical context.

The workstation is now:

• Communicating with a previously unfamiliar external destination.
• Active outside its normal operating pattern.
• Producing outbound traffic inconsistent with its established baseline.
• Exhibiting behavioral deviations following earlier unusual internal communication.

The individual network connections might not be particularly suspicious.

The behavioral sequence is.

Stage 5 – Detection and MITRE ATT&CK Context

Security Analytics identifies the deviation and generates an event.

The relevant activity can then be associated with MITRE ATT&CK context, helping the analyst understand that the behavior may relate to an exfiltration technique rather than simply representing unusual bandwidth usage.

At this point, the analyst has several useful pieces of information:

Who?
Which asset generated the suspicious activity?

Where?
Which internal or external systems were involved?

When?
Did the behavior occur outside the asset's normal activity period?

What changed?
Which aspects of the traffic deviated from the established baseline?

How significant is it?
What severity or risk has been associated with the event?

What might it represent?
Which MITRE ATT&CK tactic or technique provides context for the detected behavior?

Stage 6 – Analyst Investigation and Response

Detection is only the beginning.

The security team can now use the event information and associated flow telemetry to investigate the affected endpoint and determine whether the activity is legitimate.

Depending on the organization's incident-response processes, this could include checking:

• The identity of the user associated with the device.
• Whether the external destination is expected.
• Whether similar connections have occurred previously.
• Which internal systems the endpoint contacted before the outbound transfer.
• Whether other assets have communicated with the same destination.
• Endpoint security telemetry and authentication logs.
• Whether credentials associated with the device or user may have been compromised.

If the activity is confirmed as malicious, the endpoint can be isolated and the incident escalated according to the organization's response procedures.

The important point is that the investigation began before the attacker needed to generate an obvious signature or a massive data-transfer event.

The warning came from a change in behavior.

Why Flow Analytics Is Particularly Valuable for Security Teams

Network flow data has traditionally been associated with bandwidth monitoring, capacity planning, application visibility, and troubleshooting.

Those remain important use cases.

However, the same telemetry also describes who is communicating with whom, when communication occurs, and how traffic behavior changes over time.

That makes flow data particularly useful as another source of security visibility.

There is also an operational advantage.

Organizations may already have routers, switches, firewalls, and other devices capable of exporting NetFlow, IPFIX, sFlow, or supported flow telemetry. NetFlow Analyzer Security Analytics can therefore apply behavioral security analysis to telemetry already being generated by the infrastructure, without requiring additional endpoint agents or dedicated out-of-band sensors simply to begin collecting network behavior.

For network teams, this also creates an interesting convergence between network visibility and security visibility.

The same underlying traffic information used to troubleshoot an application or investigate bandwidth consumption can provide valuable evidence during a security incident.

Summary

Modern attacks increasingly challenge the assumption that malicious network activity will always contain an obvious malicious signature.

Once an attacker gains access to a legitimate endpoint, reconnaissance, lateral movement, and even data exfiltration can occur through traffic that appears individually legitimate.

The more useful question may therefore not be whether a particular connection is inherently malicious, but whether that connection makes sense for the asset generating it.

This is where ManageEngine NetFlow Analyzer's Security Analytics extends traditional flow monitoring.

By combining existing flow telemetry with ML-driven behavioral baselining, anomaly detection, asset-level context, risk information, and MITRE ATT&CK-aligned detections, it allows network and security teams to identify changes in behavior that conventional threshold- or signature-based monitoring may overlook.

The silent data exfiltration example demonstrates why this matters.

A small outbound transfer may not trigger a bandwidth alarm.

A new destination may not yet exist on a threat-intelligence blacklist.

An encrypted connection may reveal little about its payload.

But when a device suddenly becomes active at an unusual time, communicates with unfamiliar internal systems, establishes contact with a new external destination, and begins transferring data outside its normal behavioral profile, the combination of those deviations tells a much more important story.

For security analysts, SOC teams, and network engineers, this turns flow telemetry from a source of network statistics into another layer of actionable security intelligence.

Download ManageEngine NetFlow Analyzer and explore how Security Analytics can help identify anomalous network behavior across your environment.