Understanding packet deduplication
Deduplication is useful when multiple copies of the same packet are received, but only a single copy should be seen.
Duplicate traffic is part of any network environment and is unavoidable. However, reducing duplicate packets as much as possible helps ensure your network is more efficient. It also allows your tools to be more accurate. Duplicate packets reduce statistical accuracy, which leads to higher perceived levels of traffic or network connections. If you experience duplicate packets, consider your analytical needs and network topology when deciding whether deduplication should be used. You most often encounter them when packets are traversing multiple routers and those routers are copying their traffic to the SPAN/mirror port.
Removing duplicates from a saved packet capture can be more accurate than deduplication with the capture card. Observer has several more options than the capture card for ignoring packet header fields. These are header fields you choose to not examine (ignore) when determining if a packet is a duplicate. When all packet header fields are used as criteria (none are ignored) the capture card-based deduplication and Observer deduplication produce nearly the same results.
In some cases you may want to retain the duplicate packets. For example, when packets are being looped or when multiple VLANs are used with your hardware, you may want to keep the packets. Retaining a copy of duplicate packets and their traversal through both VLANs may be necessary when verifying whether the traffic was routed properly.
If you are attempting to find the source of duplicate packets in real time, do not deduplicate packets. Removing duplicate packets before they reach Observer or the GigaStor system lessens your ability to find the source of duplicates—if that is your goal. Instead, you can allow all duplicate packets and make changes to you monitored switches or SPANs and see if that resolves the duplicates coming in or helps locate the source.