Use monitoring resources more effectively thanks to intelligent Load Balancing

Problem

Often, analysis, monitoring and security systems have more than one port to accept and process incoming data from the corresponding network access points. Many of these systems have at least 2, 4 or even more ports ready to accept data.

Depending on the type and location of the various network access points, this offers the user the option of providing a dedicated physical port per tapped line. However, several factors are a prerequisite for this.

The speed and topology of the network lines to be analysed must be identical to the connections of the analysis system and it must be ruled out that further tap points are added in the future which are to be evaluated by the same analysis system.

Approaches

Apart from the problems with speeds and topologies, additional analysis systems can of course be installed at any time should the number of lines to be monitored increase.
However, this is often the most costly and time-consuming alternative. Besides the necessary procurement, it would mean for the user to deal with yet another system; an avoidable additional administrative effort.

To avoid such a situation, there are various options and, depending on the setup already in place, technical procedures can be used to distribute the incoming data from the measuring points more effectively to the physical ports already in place.
In many cases, it is primarily the ratio of data volume and number of measurement points to the number of available ports on the analysis system and not the basic amount of data volume that can lead to bottlenecks of a physical nature.

Both (semi-dynamic) load balancing and dynamic load balancing can help here, a feature that most network packet brokers include.
Here, a group/number of physical ports is combined on the Network Packet Broker and defined as a single logical output instance. Data streams that leave the Network Packet Broker via this port grouping are distributed to all ports that are part of this grouping, but the individual sessions remain intact.

Example

Dynamic Load Balancing Example Diagram

Let us assume the following example: 8 measuring points have been set distributed in the local network. A session between 2 end points runs via each tap point. The analysis system used is equipped with a total of 4 ports for data acquisition.
Even if one assumes that the measuring points are exclusively SPAN or mirror ports, there is still the problem that too many measuring points meet too few ports.

And this is where Network Packet Brokers with Load Balancing come into play. Load Balancing ensures that each session between 2 end points of each measuring point is sent as a whole to a single port of the connected analysis system.
For simplicity, assume that the 8 sessions of the 8 measuring points are distributed equally among the 4 ports of the analysis system, 2 sessions per port.

This is all completely dynamic and subsequently added sessions between endpoints are sent fully automatically to the ports of the analysis system that belong in the port grouping. It is not necessary to set up or change the configuration of the Network Packet Broker afterwards; the built-in automatisms allow the automated and reliable distribution of further data streams to the analysis system.

Of course, it is also possible to connect additional tap points to the Network Packet Broker and have their data included in the Load Balancing, as well as to expand the above-mentioned port grouping with additional output ports.
All these steps can be taken during operation, the additional data streams are distributed to the newly added ports of the analysis system in real time without interruptions.

Removing ports, even during operation, is no problem either! The Network Packet Broker is able to ensure that the packets/sessions are forwarded to the remaining ports of the analysis system without any loss of time or packets.

Dynamic Load Balancing Example Diagram without Network Packet Broker

Sessions & Tuples

But how can the Network Packet Broker ensure that entire sessions are always distributed on the individual ports of the load balancing group mentioned above?

For this purpose, a hash value is generated from each individual package. An integrated function ensures that in the case of bi-directional communication, the packets of both transport directions leave the Network Packet Broker again on the same port.

These hash values are determined using the so-called “5-tuple” mechanism, where each tuple represents a specific field in the header of each Ethernet frame. The available tuples on the Network Packet Broker (e.g. NEOXPacketLion), which are used for dynamic Load Balancing, are:

  • Input Port (Physical Connection)
  • Ethertype
  • Source MAC
  • Destination MAC
  • VLAN Label
  • MPLS Label
  • GTP Tunnel Endpoint Identifier
  • GTP Inner IP
  • IP Protocol
  • Source IP
  • Destination IP
  • Layer-4 Source PORT
  • Layer-4 Destination PORT

Depending on the structure and setup of the network, and depending on whether packets are transported using NAT, another very common distribution of tuples is:

  • IP Protocol
  • Source IP
  • Destination IP
  • Layer-4 Source PORT
  • Layer-4 Destination PORT

With “5-tuple” based Load Balancing, all the above-mentioned tuples are used to form a hash value which ensures that all packets, including the corresponding reverse direction, always leave the Network Packet Broker via the same port and thus, for example, the security system used always and fundamentally only receives complete sessions for evaluation.

Hash Values

In order to be able to generate the actual hash value on which the Load Balancing is based, the user has two different functions at his disposal, CRC32 and XOR.

By means of the CRC32 function, hash keys with a length of 32 bits can be generated and can be used both symmetrically and asymmetrically, while the XOR function creates a 16-bit long hash key, which, depending on the intended use, allows a higher-resolution distribution of the data, but can only output it symmetrically.

This symmetry means that even if the source IP and destination IP are swapped, as is known from regular Layer 3 conversations, the function still calculates the same hash key and thus the full Layer 3 conversations always leave the Network Packet Broker on the same physical port.

In the case of an asymmetric distribution, which is only supported by the CRC32 function, the Network Packet Broker PacketLion would calculate different hash values in the situation described above and thus also be output accordingly on different physical ports.

Dynamic Load Balancing - Screenshot - NEOXPacketLion Network Packet Broker
NEOXPacketLion Network Packet Broker – Screenshot

Dynamic Load Balancing

Another, additional function of Load Balancing is the possible dynamics with which this feature can be extended. In the case of dynamic Load Balancing, in addition to the hash value explained above, the percentage utilisation of each port that is part of the Load Balancing port group is also included in the calculation.

Of course, this procedure does not split any flows, and it also ensures that if a flow is issued based on the calculations on a specific port, this flow will always leave the Network Packet Broker via the same port in the future.

By means of a configurable timeout, the user can define when a flow loses its affiliation to an output port. In the event of a recurrence, this is then output to the participants of the load-balancing port group again in a regular manner and, both by detecting the load in the TX stream and by means of the hash value generation, it is determined which output port of the Network Packet Broker is currently most suitable for bringing the data to the connected analysis system.

Conclusion

It turns out that distributing the incoming data load by means of load balancing has been an effective way of utilising security, analysis and monitoring systems for many years. Over the years, this process has been further improved and culminates in the Dynamic Load Balancing that the PacketLion series has.

Constantly following up the configuration with regard to the distribution of the individual sessions to the connected systems is no longer necessary; this is now taken over by the intelligence of the Network Packet Broker and allows the user to use the full potential of his systems and avoid unnecessary expenditure.

Stay a the cutting edge thanks to Packet Slicing

Problem

Sling - Photo by Brands&People

Often the gap between the capacity of the recording analysis system on the one hand and the amount of incoming data on the other is so large that without appropriate additional mechanisms the analysis system is most likely not able to record all individual packets without loss.

Depending on the purpose of the analysis system, this is a major problem, as every packet counts, especially in the cyber security environment, and otherwise it is not possible to ensure that all attacks and their effects are detected.

Attacks that are not detected in time or even remain completely invisible can cause enormous damage to companies, even leading to recourse claims from possible insurers if they discover that their clients did not fulfil their duty of care.

But how does such a situation arise? It can happen very quickly that networks in companies grow, often in parallel with the business development of this company, while the often already existing analysis and monitoring systems, planned with reserves during procurement, reach the end of their reserves more and more often.
Higher bandwidths, more and more services and interfaces used in the LAN reduce the capacities to the point where the systems can no longer keep up and have to discard packets.
Photo by Lars Kienle
From this moment on, it is theoretically possible for an attacker to stay undetected in the local network, as the analysis system is hopelessly overloaded. The administrator is no longer able to see which parties in his network are talking to each other, which protocols they are using and which endpoints are being communicated with outside the LAN.

Often, however, it is not capacity problems that trigger the activation of Packet Slicing, but rather data protection reasons. Depending on where and which data is tapped and when, it may be obligatory for the company to only record and evaluate data that does not contain any personal or performance-related information.

While typically the packet header only contains connection data (WHEN, WHO, HOW, WHERE), the payload data, although usually encrypted, contains the very content data that theoretically makes it possible to measure the performance of individual users. Depending on the place of use, however, this is often neither wanted nor allowed. It must therefore be ensured that it is not possible for the administrator to reconstruct personal information from the recorded data.

Reduce analysis data by means of Packet Slicing

87 percent less thanks to Packet Slicing
And this is exactly where the “Packet Slicing” feature comes into play: with this procedure it is possible to reduce the incoming data load on your analysis system by up to 87% (with 1518 bytes packet size and Packet Slicing at 192 bytes) by simply removing the user data from each packet.

Many analysis and monitoring approaches only need the information stored in the packet header, i.e. the metadata, for their evaluations and analyses, while the user data often does not contain any important or usable information, as this is usually encrypted anyway and thus cannot be used for the evaluation.
By removing the user data, a massive relief of the processing instance is to be expected, and in some cases this enables even greater coverage of the LAN by the monitoring and analysis device.

FCS Checksum Problem


An important aspect of Packet Slicing is the recovery of the FCS checksum of each modified packet. Since the structure and length of the packet is affected by cutting away the user data, the originally calculated checksum, which was calculated by the sender and entered in the FCS field of the packet header, is no longer correct.

As soon as such a packet arrives on the analysis system, those packets are discarded or declared erroneous, as the checksum in the FCS field is still based on the original packet length. To counteract this, it is essential that the FCS checksum is recalculated and also entered for each packet from which the user data has been removed, as this would otherwise force the analysis systems to classify these packets as faulty and/or manipulated.

Network Packet Broker as a Packet Slicer

In general, there are several possibilities where the above-mentioned Packet Slicing can be activated in the visibility platform used by the customer. On the one hand, this is a case-by-case decision, on the other hand, it is also a technical one.

NEOXPacketLion - Network Packet Broker - Network Monitoring Switch | Data Monitoring Switch

Assuming that the user has set several measuring points distributed in his network, a Network Packet Brokeris often used. This device is another level of aggregation and is typically used as the last instance directly before the monitoring system. A Network Packet Broker is optically very close to a switch and enables the user to centrally combine the data from multiple measuring points (Network TAPs or SPAN ports) and send them aggregated in one or more data streams to the central analysis system.

Thus, for example, the data from 10 distributed measuring points set in 1 Gigabit lines can be sent to an analysis system with a single 10 Gigabit port by the Network Packet Broker aggregating these 1 Gigabit signals and outputting them again as a single 10 Gigabit signal.

NEOXPacketRaven Network TAP - OM4 to RJ45At this point, however, the user learns of a catch to the whole issue: often the analysis systems, although equipped with a 10Gigabit connection, are not able to process bandwidths of 10Gigabit/second as well.

This can have a variety of reasons, which, however, should not be the subject of this blog entry. However, the initial situation is predestined for the use of Packet Slicing; while one would normally have to expand one’s monitoring infrastructure at immense cost, by switching on Packet Slicing one can massively reduce the incoming flood of data and thus continue to use one’s existing systems; all that is needed is a corresponding instance with precisely this feature, which usually costs only a fraction of what would be estimated for an upgrade of the analysis systems.

Analysis Systems as Packet Slicers

Napatech High Performance Smart-NICsAnother possibility is offered to the user on the analysis systems themselves. Depending on the manufacturer, structure and components used, it is possible to directly remove the user data on the systems themselves and recalculate the checksum before the packets are passed on internally to the corresponding analysis modules.

In the vast majority of cases, an FPGA-based network card is required for this, as it must be ensured that no CPU-based resource is used to modify each individual packet. Only by means of pure hardware capacities can the user be sure that every packet is really processed accordingly; any other approach could again lead to the errors and problems mentioned at the beginning.

Packet Slicing to meet Legal Requirements

Another aspect worth mentioning is the fulfilment of legal requirements. Especially in the context of the GDPR, it may be necessary to remove the user data, as often the metadata is sufficient for an analysis.

For example, if you want to analyse VoIP, you can use Packet Slicing to ensure that unauthorised persons cannot listen to the conversation, but you can still technically evaluate the voice transmission and examine it for quality-of-service features. This allows performance values to be evaluated, privacy to be protected and legal requirements such as the GDPR to be met.

Conclusion

So we see that there are indeed different ways to distribute the final load on the analysis and monitoring systems or even, as in this example, to reduce it without losing the most important information for creating performance charts, top talkers and more. Packet Slicing is therefore a valid solution for the user, which can be easily implemented in almost all cases and achieves usable results.

Emotet Malware: Email Spoofer Awakening

According to IBM X-Force, the Emotet malware has recently been spreading in Germany and Japan, targeting companies in the area more and more aggressively.

Emotet is a banking Trojan spread by macro-enabled email attachments that contain links to malicious sites. It functions primarily as a downloader for other malware, namely the TrickBot Trojan and Ryuk ransomware. Due to its polymorphic nature, it can evade traditional signature-based detection methods, making it particularly difficult to combat. Once it has infiltrated a system, it infects running processes and connects to a remote C&C server to receive instructions, run downloads and upload stolen data (us-cert.gov).

Traditionally, Emotet has been using corporate billing notifications for disguise, often mimicking the branding of reputable institutions to make itself appear legitimate. This strategy allowed it to target victims in the USA (52 % of all attacks), Japan (22 %) and countries of the EU (japan.zdnet.com). An incident that took place in December 2019 caused the city of Frankfurt, home of the European Central Bank, to shut down its network (zdnet.com).

In Japan, however, the malware has been acting with much greater aggression compared to the past years. Increased activity was reported in late 2019, and recently, following the coronavirus outbreak in China, Emotet had a change of tactics and has now been spreading throughout Japan in the form of fake public health warnings with disturbing reports of coronavirus cases in the Gifu, Osaka and Tottori prefectures (IBM X-Force Exchange).

This is a good illustration of what makes this type of malware so dangerous – not only is it resistant to detection by signature-based methods, it also manipulates basic human emotion to disseminate itself.

Protection against Emotet therefore requires more complex measures. Besides well-informed prevention, an effective way to cope with it is to rely on behavior analysis seeking indicators of compromise (IoC). In Flowmon’s case, this takes the form of the InformationStealers behavior pattern (BPattern), which exists as a standard detection method in Flowmon ADS and describes the symptoms of Emotet’s presence in the network.

BPatterns can be thought of as a kind of description of how different malicious actors manifest themselves in the network. They allow the system to discern threats from other activity as it monitors and critically assesses the traffic. Unlike traditional signatures, BPatterns do not look for a particular piece of code and thus retain their ability to identify threats even as they transform and progress through their life-cycle.

According to an analysis published by Fortinet, Emotet uses 5 URLs to download payload and 61 hard-coded C&C servers (fortinet.com/blog). This information is included in the BPattern, and is used by the system to recognize the infection and contain it before it can spread. For an added layer of protection, there is a BPattern for TrickBot as well (TOR_Malware). Both patterns are periodically updated depending on how the Trojans evolve and are delivered to users as part of regular updates. It was Flowmon’s partner Orizon Systems that alerted us to the increased incidence of the Emotet malware and prompted the most recent update.

But no protection is infallible and everyone is advised to keep several layers of cyber protection in place and up to date – including antivirus, IoC detection on firewalls, intrusion detection systems (IDS) and behavioral analysis on the network. Because Emotet spreads by spoofed email, users should exercise caution opening attachments, especially those who come in daily contact with bills and documents from outside parties, and report any suspicious or unusual email to the security team.

To learn more about threat detection using Flowmon ADS, contact us for more information or try a demo.

The Effect of Packet Loss on an IDS Deployment

At SuriCon 2019, Eric Leblond and Peter Manev – both of whom are key contributors in the Suricata community – presented important test results, emphasizing the implications of packet loss. Let’s dig a little deeper into the importance of zero packet loss in an IDS deployment.

The effect of packet loss on a variety of network analysis gear varies widely based on the function the analysis device is performing. The measurement accuracy of network and/or application performance monitoring devices is affected when packets are dropped by the network sensor. In the case of a cybersecurity appliance, like a Suricata-based Intrusion Detection System (IDS), missed intrusion alerts are directly affected by system packet loss. This is also the case for file extraction.

The effect of packet loss on intrusion alert generation

No matter how good the IDS rule, if all packets for a given session are not delivered to the IDS, alerts can be missed. This is mainly due to how an IDS processes a network session. The given packets that are dropped within the session will determine if the IDS has enough data to generate an alert. In many cases, the IDS will drop the entire session if key packets are missing. A missed alert could mean that an intrusion attempt went undetected.

To measure the missed alerts, the following methodology was used:

  • Traffic source is a PCAP file containing actual network traffic with malicious activity
  • PCAP file is processed to simulate specified random packet loss
  • Suricata alerts are compared for original PCAP file to modified file

Sample numbers:

  • 10% missed alerts with 3% packets loss
  • 50% missed alerts with 25% packets loss

The effect of packet loss on IDS file extraction

Part of deploying a successful IDS strategy is to also automate file extraction. Most IDS engines support HTTP, SMTP, FTP, NFS, and SMB protocols. The file extractor runs on top of the protocol parser, which handles dechunking and unzipping the request and/or any response data if necessary. Loss of a single packet for a network stream carrying a file in most cases will cause file extraction to fail.

Sample numbers:

  • 10% failed file extraction with 4% packet loss
  • 50% failed file extraction with 5.5% packet loss

In conclusion, the test results show how important zero packet loss is to a successful IDS deployment. FPGA SmartNIC features like on board burst buffering, DMA, and optimized PCI express performance will minimize or completely eliminate packet loss in a standard server-based IDS.

Ensuring performance resilience with deduplication

Performance resilience is the ability to ensure the performance of your commercial or home-made appliance in any data center environment. In other words, to ensure that your performance monitoring, cybersecurity or forensics appliance is resilient to common data center issues, such as badly configured networks, inability to specify desired connection type, time sync, power, space, etc.

 

In this blog, we will look at deduplication and how support of deduplication in your SmartNIC ensures performance resilience when data center environments are not configured properly – router and switch SPAN ports specifically.

 

Assume the worst

 

When designing an appliance to analyze network data for monitoring performance, cybersecurity or forensics, it is natural to assume that the environments where your appliance will be deployed are configured correctly and adhere to best practices. It is also fair to assume that you can get the access and connectivity you need. Why would someone go to the trouble of paying for a commercial appliance or even fund the development of an appliance in-house, if they wouldn’t also ensure that the environment meets minimum requirements?

 

Unfortunately, it is not always like that, as many veterans of appliance installments will tell you. This is because the team responsible for deploying the appliance is not always the team responsible for running the data center. Appliances are not their first priority. So, what happens in practice, is that the team deploying the appliance is told to install the appliance in a specific location with specific connectivity, and that is that. You might prefer to use a tap, but that might not be available, so you need to use a Switched Port Analyzer (SPAN) port from a switch or router for access to network data.

 

While this might seem acceptable, it can lead to some unexpected and unwanted behavior that is responsible for those grey hairs on the heads of veterans! An example of this unwanted behavior is duplicate network packets.

 

How do duplicate packets occur?

 

Ideally, when performing network monitoring and analysis, you would like to use a tap to get direct access to the real data in real time. However, as we stated above, you can’t always dictate that and sometimes have to settle for connectivity to a SPAN port.

 

The difference between a tap and a SPAN port is that a tap is a physical device that is installed in the middle of the communication link so that all traffic passes through the tap and is copied to the appliance. Conversely, a SPAN port on a switch or router receives copies of all data passing through the switch, which can then be made available to the appliance through the SPAN port.

 

When configured properly, a SPAN port works just fine. Modern routers and switches have become better at ensuring that the data provided by SPAN ports is reliable. However, SPAN ports can be configured in a manner that leads to duplicate packets. In some cases, where SPAN ports are misconfigured, up to 50% of the packets provided by the SPAN port can be duplicates.

 

So, how does this occur? What you need to understand with respect to SPAN ports is that when a packet enters the switch on an ingress port, a copy is created – and when it leaves a switch on an egress port, another copy is created. In this case, duplicates are unavoidable. But it is possible to configure the SPAN to only create copies on ingress or egress from the switch, thus avoiding duplicates.

 

Nevertheless, it is not uncommon to arrive in a data center environment where SPAN ports are misconfigured and nobody has permission to change the configuration on the switch or router. In other words, there will be duplicates and you just have to live with it!

 

What is the impact of duplicates?

 

Duplicates can cause a lot of issues. The obvious issue is that double the amount of data requires double the amount of processing power, memory, power, etc. However, the main issue is false positives: errors that are not really errors or threats that are not really threats. One common way that duplicates affect analysis is by an increase in TCP out-of-order or retransmission warnings. Debugging these issues takes a lot of time, usually time that an overworked, understaffed network operations or security team does not have. In addition, any analysis performed on the basis of this information is probably not reliable, so this only exacerbates the issue.

 

How to achieve resilience

 

With deduplication built-in via a SmartNIC in the appliance, it is possible to detect up to 99.99% of duplicate packets produced by SPAN ports. Similar functionality is available on packet brokers, but for a sizeable extra license fee. On Napatech SmartNICs, this is just one of several powerful features delivered at no extra charge.

 

The solution is ideal for situations where the appliance is connected directly to a SPAN port, dramatically reducing the amount of damage that duplicates can cause. But, it also means that the appliance is resilient to any SPAN misconfigurations or other network architectural issues that can give rise to duplicates – without relying on other costly solutions, such as packet brokers, to provide the necessary functionality to complete the solution.