Axtrinet™ APG Ethernet Packet Generators offer affordable 40Gbps & 10Gbps full wire-speed Ethernet load generation, capture and analysis capabilities for R&D, manufacturing, sales and support teams developing and selling products with high speed Ethernet interfaces.

An intuitive Graphical Control Interface or TCL-scripted interface can be used to configure and control the packet generation, capture and analysis capabilities of the unit via a Linux or Windows PC, managing it locally over USB or remotely over Ethernet LAN.

This application note describes an example of how the stream generation capabilities may be used to diagnose network equipment performance issues.

Network Performance Issues

A customer was observing packet retransmissions that were causing poor performance on their test network. The problem was traced to a network switch dropping occasional packets.

Further analysis of the network traffic showed that the packet loss occurred immediately after a burst of line-rate short packets following a long packet. This behaviour is known as ‘head of line blocking’, and is known switch behaviour on congested networks. However, the observed packet loss rate was significantly higher than expected, and was indicative of egress port buffering problems.

Having identified the likely cause, the behaviour had to be recreated and categorised by the R&D team before updating the switch configuration in the embedded software.

Axtrinet™ APG Packet Generating Capabilities

Axtrinet™ APG Ethernet Packet Generators offer 8 parallel independent configurable Ethernet stream generators per port.

A transmit stream generates a controlled number of Ethernet frames at a controlled length and rate; with a fixed header configuration, fixed or varying header contents, and a fixed or varying payload.

The stream transmit rate defines how the packets are generated; either continuously, or as a single or multiple burst containing a fixed number of packets.

The packet length can be set to fixed, or incrementing, decrementing or random over a byte range.

Each transmitted packet can be optionally marked with a 32-bit port signature; a 32-bit sequence number and a 64-bit transmit timestamp, to an accuracy of ±8ns.

Streams are configuration using the simple to use Graphical Control Interface.

Transmit control allows the packet transmission to be started, stopped or stepped; and the port counters reset.

Transmit packet and byte, and receive packet, byte and error counts give an easily visible indication of any packet errors.

Axtrinet™ APG Ethernet Packet Generators offer a dynamically allocated 1GByte ‘deep’ packet capture buffer, and a fixed 64KByte capture buffer per port.

All packets captured by the Axtrinet™ APG Ethernet Packet Generator are timestamped with the arrival time at the port.

The captured data can be downloaded to the Control Interface GUI for further analysis, or stored externally as a PCAP file for offline analysis in a third-party tool, such as Wireshark™.

The Control Interface displays:

• the captured packet data
• packet lengths
• inter-packet gap (IPG), derived from the receive timestamps, and
• latency, derived from the transmit and receive timestamps for each packet.

Diagnosis

Uni-directional tests were initially performed on the switch. The transmit port was configured with two active streams, each configured to generate continuous fixed-length line-rate packets, with MAC headers and random payload.

The first stream was configured to generate 1514 byte packets at 1% utilisation; with the second stream generating 64 byte packets at 95% utilisation.

The received data stream was captured in the 1GByte buffer. By analysing the packet counters to verify packet loss; captured inter-packet gaps (IPG) to determine retransmission timing; and packet sequence numbers to identify dropped packets, the switch buffer characteristics could be determined.

The testing demonstrated packet loss occurred after 8 short packets, significantly lower than expected.

Analysis of the embedded software verified that the switch configuration was incorrect, and that the measured behaviour was consistent with the misconfiguration.

Resolution & Further Testing

An Ethernet switch has a pool of buffers available for ingress and egress buffering. A single port can be configured to use more than its fair share of buffers; and although the total number of configured buffers may exceed the actual number of buffers available, it does allow the switch to handle ‘normal’ bursty network traffic.

Increasing the buffer allocation allowed the switch to support jumbo frames without dropping short packets. Configuring the Axtrinet™ APG to generate 16000 byte packets verified correct behaviour with a revised switch configuration.

Further testing was performed with random sized 1500-16000 byte ‘long’ packets and 64-100 byte ‘short’ packets at different rates to characterise behaviour on one port.

The testing was extended to bi-directional traffic; and then multi-port testing to further characterise the switch behaviour.

After characterising the behaviour under laboratory conditions, a recommended switch buffer allocation was implemented in the embedded software by the customer.

Extensive testing on the live network verified that the packet loss and retransmissions had stopped.