Understanding GigaStor indexing
This section describes how the GigaStor captures packets and indexes them for statistics.
Indexing is an important part of how the GigaStor is able to be as efficient as it is. A brief synopsis of indexing in the GigaStor is this:
All captured packets are written to disk. None of the settings in the GigaStor Control Panel control what is written to disk in any way.
Indexing is not used for packet capture. It is only for statistics.
GigaStor Control Panel > Settings > Capture and Analysis options tells the GigaStor which packets to index for statistics.
GigaStor Control Panel > Settings > Collect and Show GigaStor Indexing Information by tells GigaStor how many entries it can use every 15 seconds. After the maximum number of entries for a 15 second period is reached, new data that was not already being indexed is not indexed for that 15 second period; however, packets that were already being indexed continue to be indexed during that 15 second period.
Every 15 seconds the GigaStor writes all indexed data for 15 second interval that was just indexed. The indexed data is cleared from memory and indexing of the next 15 seconds begins.
Previously indexed data has no effect on any other 15 second interval, except for the need to see the SYN-SYN/ACK-ACK to begin collecting “new” server data. This means that if in one 15 second period the maximum number of entries was reached and a new conversation is started that continues into the next 15 second interval, there is nothing that prevents the subsequent 15 second interval from beginning to index the new conversation that was not indexed in the previous 15 second interval.
When GigaStor Control Panel > Settings > Enable Intelligent TCP protocol determination is enabled, a SYN-SYN/ACK-ACK is required. After the GigaStor sees a SYN-SYN/ACK-ACK for a server, it no longer needs to see the SYN-SYN/ACK-ACK to collect data from that server on the port that it saw the SYN-SYN/ACK-ACK. If for any reason the GigaStor probe is not running, it needs to see the SYN-SYN/ACK-ACK to index data. If GigaStor Control Panel > Settings > Enable Intelligent TCP protocol determination is unchecked, the GigaStor does not need to see the SYN-SYN/ACK-ACK to ever index data.
For more details about indexing in the GigaStor continue reading the rest of this section.
Every 15 seconds the GigaStor writes indexed, statistical data into a GigaStor.ometa file on the D: drive. It contains only statistical (indexed) information “collected” by the GigaStor. This file and the statistics it contains have no relationship to what packets are written to disk. When the capture card sees any packet, it is immediately timestamped and passed to the GigaStor buffer. The GigaStor writes all packet data to disk regardless of whether a packet is indexed.
Also on the D:\ drive are a number of .odat files. These files contain the actual packets that are written to disk and used for analyzing.
The GigaStor does not index every single packet. There are a number of factors that result in a packet not being indexed. Anything you see in the GigaStor Control Panel should be used for reference, not as absolute numbers. For absolute numbers, you must analyze the packets and view them in the Decode pane.
At the beginning of each 15 second period (the 15 seconds is not based on the system time clock period, but on timestamps from the captured packets) the GigaStor takes all of the indexed data that it has and writes it into the GigaStor.ometa file. The GigaStor then clears out the statistical memory that was used for indexing during the 15 second collection period and begins analyzing the next 15 seconds for statistical data. After a packet has been analyzed, it is written to disk. If for some reason a packet is skipped, it is written to disk before the next packet is analyzed.
Again, not every packet is indexed. This does not mean that if a packet is not indexed, that it is not written to disk. The GigaStor writes every packet to disk, even if it is not indexed. If there are 1,000,000 packets that come in during a 15 second period, and the GigaStor only analyzes 85,000, it will still writes 1,000,000 packets to the hard drive.
If the Screen Resolution in the GigaStor Control Panel is set to less than 15 seconds, the GigaStor does not use the index file (GigaStor.ometa) to see what was indexed because the time frame is smaller than 15 seconds. Instead it reads the data that is written to disk in the .odat file to produce the reports and not the indexed data.
The indexed, statistical information that comes from the indexed data is not 100% accurate when compared to packet capture. More importantly, it is not intended to be. It is, however, statistically accurate.
When the GigaStor attempts to analyze a packet to index, it does not analyze the packet if the GigaStor sampling ratio causes that packet to be excluded from metadata. GigaStor will skip that packet for GigaStor statistical metadata, but will still write the packet itself to disk as well as track its associated indexing information for GigaStor accelerated analysis.
After 15 seconds, the GigaStor starts over, so everything is cleared out and it all starts from zero entries per index data table, but the GigaStor does keep track of which devices it classified as servers. For instance, if in one 15 second period, the GigaStor sees a SYN-SYN/ACK-ACK and determines that port 8080 on 10.0.0.1 on is a server, in the next 15 second period, the GigaStor does not require a SYN-SYN/ACK-ACK to know that port 8080 on 10.0.0.1 is a server. It already knows and continues indexing any 10.0.0.1 8080 as the server. The indexing of server 10.0.0.1 on port 8080 requires that either you establish 8080 as a known protocol or you have disabled the GigaStor Control Panel > Settings > Intelligent TCP Determination option. However, depending on which options are enabled and disabled, the GigaStor may completely ignore 10.0.0.1 on 8080 from being indexed.