Short for network statistics, netstat is a utility that is built into most Windows and Linux clients. It provides an array of statistics on current ports, routing, IP stats for transport layer protocols, and serves as a forensic tool to link processes with network performance while digging deeper into the current network status. The netstat utility bundles several actions into a single command, with different options available. Because netstat is multiplatform, the syntax is similar across platforms with slight variations.

Standard netstat

In its standard form without any arguments, netstat provides an overview of the current network status broken out by each connection or socket. Each column displays the following information:

• Proto: Protocol of the active connection. The protocol could be TCP or UDP and has a 4 or 6 associated, specifying if it is IPv4 or IPv6, respectively.
• Recv-Q and Send-Q: Value of the receiving and sending queue in bytes. Nonzero values specify the number of bytes in the queue that are awaiting to be processed. The preferred value is zero. If several connections have nonzero values, this implies something is causing processing to be delayed.
• Local Address and Foreign Address: Lists the hosts and ports the sockets are connected with. Some of these are local connections to the host itself.
• State: Displays the current state of the TCP connection, based on the TCP protocol. Because UDP is a stateless protocol, the State column will be blank for UDP connections. The most common TCP states include:

• Listen: Waiting for an external device to establish a connection.
• Established: Ready for communication on this connection.
• Close Wait: The remote machine has closed the connection, but the local device has not closed the connection yet.
• Time Wait: The local machine is waiting for a period of time after sending an ACK to close a connection.

For more information about the states of a TCP connection, see RFC 793.

Figure 40.    netstat

The netstat utility reveals a lot of information about the status of network connections, and it also provides information for a more thorough forensic analysis. Based on the output from netstat, some of the scenarios can be generalized. For example:

• Recv-Q has a value greater than zero but is in a Close Wait state. This indicates that these sockets should be torn down but are hanging. If several sockets are in this state, the application might be having difficulty tearing down the connection, warranting additional investigation.
• Connections that have localhost as the Local and Foreign address denote an internal process using the TCP stack to communicate. These connections are not concerning and are standard practice.

netstat -s -p tcp

The netstat utility offers several options, but the -s provides statistics by protocol and -p displays the net to media tables. These options reveal current health, and tcp limits it to the TCP protocol. The following figure shows a sample output of this command with the areas to examine highlighted in red.

Figure 41.    netstat -s -p tcp

The fields highlighted in red in the preceding figure must be reviewed as a ratio of the total packets that are transmitted and received as a percentage. Also, these statistics should be monitored for sudden increments. As a guideline, under 1 percent is not concerning, but this also depends on the workload. The fields highlighted in the preceding figure provide the following information:

• Retransmitted packets: Packets that are retransmitted consume network bandwidth and could be the reason for further investigation. However, examining the percentage is critical. In this case, 249379 out of 235829612 were retransmitted, which is 0.105 percent.
• Duplicate acknowledgements: High latency between endpoints might lead to duplicate acknowledgments, but the ratio must be examined. In this case, it is 0.419 percent. This number varies, depending on the workload.
• Out-of-order packets: Out-of-order packets are placed in order by TCP before being presented to the application layer. This activity affects the CPU and overall stack performance because of the additional effort involved in analyzing the packets. Performance is most affected when packets arrive out of order with a significant time gap or when several packets are out of order. The ratio, in this case, is 0.197 percent, which is negligible.

netstat -i

Another option for netstat is the –i option, which is the interface display, listing cumulative statistics for total packets transferred, errors, MTU, and collisions by the interface. Because netstat –i lists all available interfaces, the back-end, and front-end interfaces are displayed. The following figure shows a sample output of netstat –i with the –h option, making it easier to interpret:

Figure 42.    netstat -i

From the output shown in the preceding figure, netstat –i lists the following columns:

• Name: Network Interface Card (NIC) name. Loopback interfaces are listed as lo0, and ib specifies InfiniBand.
• MTU: Lists the MTU specified for the interface.
• Network: Specifies the network associated with the interface.
• Address: MAC address of the interface.
• Ipkts: Input packets are the total number of packets received by this interface.
• Ierrs: Input errors are the number of errors reported by the interface when processing the Ipkts. These errors include malformed packets, buffer space limitation, checksum errors, errors generated by media, and resource limitation errors. Media errors are errors specific to the physical layer, such as the NIC, connection, cabling, or switch port. Resource limitation errors are generated at peak traffic when interface resources are exceeded by usage.
• Idrop: Input drops are the number of packets that were received, but not processed and therefore dropped on the wire. Dropped packets typically occur during heavy load.
• Opkts: Output packets are the total number of packets transmitted by this interface
• Oerrs: Output errors are the number of errors reported by the interface when processing the Opkts. Examples of errors include the output queue reaching limits or an issue with the host.
• Coll: Collisions are the number of packet collisions that are reported. Collisions typically occur during a duplex mismatch or during high network utilization.

In general, errors and dropped packets require closer examination. However, as noted in the previous netstat section, the percentage of errors and percentage of dropped packets are the main factors. For further analysis, consider:

• Ierrs should typically be less than 1 percent of the total Ipkts. If greater than 1 percent, check netstat –m for buffer issues and consider increasing the receive buffers. Before implementing changes on a production system, test buffer changes in a lab environment. See PowerScale network stack tuning for additional details.
• Oerrs should typically be less than 1 percent of the total Opkts. If greater than 1 percent, it could be a result of network saturation, otherwise consider increasing the send queue size.
• The ratio of Coll to Opkts should typically be less than 10 percent. If greater than 10 percent, it could be a result of high network utilization.

netstat -m

The netstat –m option displays the status of network memory requests as mbuf clusters. netstat –m is a powerful option for a complete forensic analysis when one of the other netstat commands previously mentioned raises concern. If mbufs are exhausted, the node cannot accept any additional network traffic.

Figure 43.    netstat -m

The netstat –m output provides information in regard to available and used mbufs. The area highlighted in red confirms if any memory requests have been denied. In the preceding example, a quick glance at this area reveals that no requests have been denied.

For more information about netstat options, see the FreeBSD manual netstat page at https://www.freebsd.org/cgi/man.cgi?query=netstat&manpath=SuSE+Linux/i386+11.3.