Understanding duplicate packets
Duplicate packets lower the statistical accuracy of analysis, increase network link saturation, and can interfere with tools. Packet deduplication removes duplicate packets and helps you avoid those situations.
 
A duplicate packet is any packet that is identical to another packet. The packet header is inspected and all fields must be identical for it to be a duplicate. However, there are some situations where the header has been modified slightly during the packet's journey. These situations require some fine-tuning of the deduplication settings to ignore those fields that were modified before the duplicate packet is received.
Duplicate packets are tracked on a per connection basis and within a certain window. If two identical packets are received on two different network ports, they are tracked and one is marked as a duplicate as long as both packets pass through the same connection. Also, the Matrix can recognize a packet as being a duplicate if it is identical to another packet within 600 milliseconds or 6000 packets, whichever comes first. Any packet that falls outside of that range is considered unique to ensure throughput for your network.
 
Understanding packet deduplication
Deduplication is useful when multiple copies of the same packet are received, but only a single copy should be seen and forwarded out the tool ports.
 
Duplicate traffic is part of any network environment and is unavoidable. However, reducing duplicate packets as much as possible helps ensure your network is more efficient. It also allows your tools to be more accurate. Duplicate packets reduce statistical accuracy, which leads to higher perceived levels of traffic or network connections. If you experience duplicate packets, consider your analytical needs and network topology when deciding whether deduplication should be used. You most often encounter them when packets are traversing multiple routers and those routers are copying their traffic to the SPAN/mirror port.
In some cases you may want to retain the duplicate packets. For example, when packets are being looped or when multiple VLANs are used with your hardware, you may want to keep the packets. Retaining a copy of duplicate packets and their traversal through both VLANs may be necessary when verifying whether the traffic was routed properly.
If you are attempting to find the source of duplicate packets in real time, do not deduplicate packets. Removing duplicate packets before they reach Observer or the Observer GigaStor system lessens your ability to find the source of duplicates—if that is your goal. Instead, you can allow all duplicate packets and make changes to you monitored switches or SPANs and see if that resolves the duplicates coming in or helps locate the source.
 
Scenario 1: Receiving network traffic from multiple routers
Cause of duplicates: Some packets are traversing multiple routers and those routers are copying their traffic to the SPAN/mirror port. When this occurs it causes duplicate packets in the Matrix.
Non-duplicate fields: Not only is each router decrementing the TTL field in the IP header, but it is also modifying the MAC address.
Solution: Ignore the MAC address pair and TTL fields.
Scenario 2: Receiving network traffic from multiple VLANs
 
Cause of duplicates: Some packets are traversing some of your VLANs. If the SPAN/mirror port is configured to copy packets traversing each VLAN, any packets that travel through multiple VLANs are duplicated.
Non-duplicate fields: In the Ethernet header, the Ethertype field may change if the packet is not encapsulated with a VLAN header when the packet is copied. If both packets contain a VLAN encapsulation header, then the VLAN values will differ. It is possible the TTL field may also differ, and in some situations, the MAC address pair may have changed.
Solution: Ignore the Ethertype and VLAN/MPLS fields, and it may also be necessary to ignore the TTL and the MAC address pair fields.
 
How to deduplicate packets
You can remove duplicate packets that reach the Matrix. This ensures that tool ports only send unique packets to analysis tools, increasing the accuracy and efficiency of analysis.
Packet deduplication requires two steps:
1. Enable packet deduplication in a rule.
2. Direct the Matrix to identify duplicate packets.
 
 
How to direct the Matrix to identify duplicate packets
You must direct the Matrix which packet fields to ignore when determining duplicate packets. For example, doing so ensures that packets with different Time to Live (TTL) values—yet are otherwise identical—are deduplicated.
The packet fields to ignore, for determining duplicate packets, are configurable in the layout properties. Unlike other settings, these settings affect the entire layout because the hardware-accelerated deduplication engines in the Matrix must work in parallel.
To direct the Matrix to identify duplicate packets according to your definition, complete the following steps:
1. Starting in the dashboard, click Ports.
The layout designer appears, where connections between network and tool ports can be created.
2. Click Properties.
3. In the Deduplicate Ignored Fields area, select which fields to ignore.
If selected, the criteria is ignored and not evaluated when determining duplicate packets.
4. Click OK
 
You successfully directed the Matrix to identify duplicate packets according to your definition. Remember, these settings affect the deduplication behavior of the entire layout, but packet deduplication is still enabled and disabled in individual rules.
 
How to enable packet deduplication in a rule
In a rule, you can enable packet deduplication. Any duplicate ingress packets, coming from network ports connected to the rule, are removed before being forwarded to tool ports and ultimately your analysis tools.
Prerequisite(s):  
These steps require that at least one rule exists in your rules library. You can create a new rule if necessary.
 
While duplicate packets are determined by how your layout properties are configured, the actual implementation of the packet deduplication feature is still controlled on a per-rule basis. This ensures that individual rules remain the deciding factor if packet deduplication is enabled or not. When packet deduplication is enabled in a rule, however, the deduplication behavior is always controlled by the layout the rule is used in.
To enable packet deduplication in a rule:
1. Starting in the dashboard, click Rules.
The rules and filters designer appears, where rules and filters can be created and edited.
2. Ensure the Rules tab is selected.
3. Click a rule from the list.
The rule opens and is ready to edit.
4. Select Deduplicate.
If selected, hardware-accelerated packet deduplication removes duplicate ingress packets in real time.
5. Click Save.
 
Packet deduplication is now enabled in the rule. Connecting this rule between network and tool ports causes the removal of duplicate ingress packets. Due to this, no duplicate packets are forwarded to tool ports, and therefore none are forwarded to your analysis tools.