Final report of ITS Center project: Wireless Testing Protocols

A Research Project Report

For the Center for ITS Implementation Research

A U.S. DOT University Transportation Center

WIRELESS TESTING PROTOCOLS

Principal Investigator

Dr. Aaron Schroeder

Virginia Tech Transportation Institute
3500 Transportation Research Plaza (0536)
Blacksburg VA 24061
Phone: 540-231-1505
Fax: 540-231-1555

2006

Disclaimer

The contents of this report reflect the views of the authors, who are responsible for the facts and the accuracy of the information presented herein. This document is disseminated under the sponsorship of the Department of Transportation, University Transportation Centers Program, in the interest of information exchange. The U.S. Government assumes no liability for the contents or use thereof.

Wireless Testing Protocols

Dr. Aaron Schroeder

Introduction

IP based wireless network usage is growing significantly among DOTs. Agencies are realizing that wireless links are an additional communications medium that can be added to their existing toolbox of services. Wireless can be used as a temporary solution or as a permanent one for certain areas where fiber or other wireline service is not realistic. As the use of wireless IP systems grows, agencies need methods for evaluating and comparing systems and architectures.

This project’s objective was to develop hardware and software tools that can be used to quantitatively measure bandwidth metrics across wireless networks. These tools will either transmit real data traffic across a wireless network and measure the amount of data transmitted over time, or will simulate different types of traffic and estimate the bandwidth.

Wireless Internet devices are proving to be a viable alternative to traditional communications mediums for Departments of Transportation (DOTs) and other public agencies. These devices are used to transport data from low-level Road Weather Information Systems (RWIS) to high-bandwidth streaming video. The useable bandwidth across a wireless link is one of the most important metrics for evaluating and comparing wireless systems. This performance specification will drive the quantity and types of devices that the wireless system can adequately accommodate.

A repeatable methodology is required to accurately measure the Internet Protocols that are used for transferring data. These protocols include File Transfer Protocol (FTP), Internet Control Message Protocol (ICMP), Transmission Control Protocol (TCP), and User Datagram Protocol (UDP). TCP and UDP are most often used when assessing the true available bandwidth, often referred to as payload bandwidth, across wireless links.

Why Bandwidth

When assessing wireless networks there are many parameters that can be investigated. Many focus strictly on the wireless aspect when comparing and evaluating systems. Metrics such as signal-to-noise ratio (SNR), bit error rate (BER), and received signal strength indication (RSSI) are some of the parameters used in wireless technologies. These metrics are all important to the performance of a wireless network; however for this project it was decided that the useable bandwidth of a system is the most telltale metric for comparing systems.

The wireless systems the Virginia Tech Transportation Institute (VTTI) has been investigating are all Internet Protocol (IP) based systems. These are similar to an Ethernet system but use a wireless medium instead of a wired medium. The applications that an end user would use such as video or data are all dependent on the useable bandwidth that the system is capable of providing.

Wireless parameters such as SNR, BER, and RSSI all have an end effect on the bandwidth. Low signal strength or a large number of bit errors correlate to lower bandwidth. Therefore, the researchers opted to analyze bandwidth performances as an overall indicator of system performance.

Network Simulation Tools

One way to test the bandwidth capacity of a system is to simulate different types of data traffic across the network. Systems will react differently to the type of data traffic that is being sent across it. For example a FTP data stream may work on a particular architecture better than a streaming media transfer.

There are several network simulation products on the market that will operate across wireless medium. Chariot (Figure 1) by IXIA software was chosen to test on our systems.

Chariot provides a nice, easy-to-use interface and is fairly powerful in the types of simulation that it can perform.

Figure 1 IXIA Chariot Front End

In order to use Chariot, a special software program call an Endpoint must be placed at each node of the network where bandwidth measurement is required. An endpoint can be placed on the console computer that runs the simulation or on a stand-alone computer co-located with the wireless node.

VTTI researchers custom built single-board computers (Figure 2) that run on a LINUX operating system. Chariot endpoints were placed on these computers and placed in the field with the wireless nodes.

Figure 2 Custom Made Single-Board Computer

These miniature computers run a LINUX operating system and are IP addressable just like any network device such as an IP camera. Chariot uses these endpoints as nodes to initiate data transfers or to receive data transfers. Chariot allows up to 10 simultaneous data streams to be simulated at the same time. The data streams use scripts that simulate a variety of different types of traffic such as streaming video or file transfers.

The graphical output provides bandwidth measurements over time.

Chariot has proven to be a good tool for measuring bandwidth performance. It can simulate a variety of traffic including FTP, streaming media, and voice over IP (VOIP). VTTI has used it to measure UDP and TCP types of data transfer.

How Chariot is Used

Typically, the wireless systems tested follow a serial architecture (Figure 3).

Figure 3 Typical Serial Wireless Architecture

A serial architecture is a series of single point-to-point to links tied together. One of the major issues with serial systems is the bandwidth degradation that can occur over successive hops (Figure 4). VTTI uses tools like Chariot to quantify the throughput across one hop, two hops, three hops, etc…

Figure 4 Degradation of Throughput Across Multiple Hops

Typically each node is designated with a number: 1, 2, 3, 4, 5. Chariot tests are run between single nodes such as Node 1 to Node 2, or Node 3 to Node 4. To test the bandwidth across two successive hops a test is run between Node 1, Node 2 and Node 4, and Node 3 and Node 5. Tests for 3 hops can be run between Node 1 and Node 4. Finally, a 4-hop test can be run between Node 1 and Node 5.

In addition, tests can be run in two directions where the first node acts as the initiator and sends data to the receiving node. Typically each pair of nodes is tested in both directions.

Limitations of Network Simulation Tools

Network simulation tools make measuring bandwidth on IP-based systems realistic. However, there are some drawbacks. One drawback is that each simulation tool uses its own proprietary protocol for calculating the bandwidth measurement. It is difficult to say how accurate each methodology is. What is more important is the repeatability of the measurement process. The same simulation can be run across different technologies and architectures and can provide insight into the differences between systems. .

A specific area of concern is the methodology used to calculate UDP throughput. By definition UDP is a connection-less protocol. There is no data handshaking that occurs as with TCP. Chariot uses a custom protocol to measure any data losses when transferring files. The researchers had some concern that the protocol used to measure loss in UDP was skewing the results. Again, this is an accuracy issue; however the researchers are confident of the repeatability of the UDP measurements.

In addition, there is concern about how well network simulation products handle timeouts associated with mobile wireless environments. In mobile hand-off situations, wireless connectivity is lost when a client passes out of range of an Access point and into range of the next access point. During this down time there is also no network connectivity. When the client regains wireless connectivity with the next Access Point, networking can potentially resume. However, if a network simulation was running during the handoff, even though wireless operation has resumed, there seems to be a longer delay in restarting the simulation (Figure 5).

Figure 5 Chariot Output Indicating Mobile Handoff

Therefore the simulation results do not correctly show the actual length of the latency between handoffs. Latency measurement is extremely important in mobile situations and the results from Chariot cannot be used to quantify this metric.

Chariot also provides the throughput average which is the metric that is often used to characterize a system. The problem is that the average is made up of many peaks and valleys. In addition, tests sometimes start out at high throughput and then drop to a lower throughput later in the test. In this case, just the average value does not provide insight into the actual performance of the link.

Chariot allows the user to change many variables. Variables such as window sizes, packet sizes, timing records, and other run options provide a fair amount of adjustability for tests. Throughput can be optimized for certain systems by changing some of these variables. However the user must realize that changing some of these variables might not be indicative of real world applications.

Tools with Higher Time Precision During Handoffs

As discussed earlier, network simulation tools provide a poor time resolution during handoffs. It is usually impossible to accurately quantify the handoff latency using tools such as Chariot that experience a ‘shutdown’ during the wireless handoff.

A more accurate approach is to use Ping tools that restart their operation as soon as networking has been restored after a wireless handoff. A Ping can be run from any DOS command window. The Ping utility that comes with windows works fairly well, however the time steps between sent pings is fixed. If the latency handoff is smaller than the fixed time step of the ping utility then it is impossible to accurately quantify the handoff latency.

More sophisticated ping utilities than the standard command line PING exist that provide the user with greater control over variables such as time steps between ping requests. VTTI has used a utility called HRPING. HRPING allows the user to set time resolution for ping requests as low as several milliseconds.

HRPING outputs look just like regular ping outputs except when a timeout occurs the next ping request is indicated in capital letters. This allows the user to pinpoint periods of latency. Each time step is indicated with a hexadecimal tag. The user is required to manually convert the tag into a decimal number and then count the number of tags that are missing between two ping responses. The number of tags missing multiplied by the user selected time step between ping requests indicates the latency.

Reply from 101.1.1.1: seq=0063 time=9.985ms TTL=254 ID=f091

Reply from 101.1.1.1: seq=0064 time=4.698ms TTL=254 ID=f092

Reply from 101.1.1.1: seq=0065 time=9.540ms TTL=254 ID=f093

Reply from 101.1.1.1: SEQ=007c time=80.561ms TTL=254 ID=f0ab

Reply from 101.1.1.1: seq=007d time=9.838ms TTL=254 ID=f0ac

The above output from HRPING was created with a time interval of 100 ms between sequences. When a handoff occurs, HRPING indicates the restart of networking via a capitalized notation. This allows us to easily pinpoint where handoffs occur. By counting the number of missing sequences (indicated in hexadecimal) researchers can estimate the duration of the handoff.

Conclusions

A reliable, repeatable methodology is required for comparing wireless networks. Agencies need tools in order to compare technologies and evaluate deployments. The methodologies developed in this project provide some options for DOTs and other agencies.

Network simulation tools provide a straightforward method for quantifying the performance of wireless links across single and multiple hops. They do have some shortcomings in terms of mobile handoffs and how they measure UDP throughputs.

Ping utilities such as HRPING provide a very high time resolution in order to properly evaluate wireless handoffs. The current tools are not the most user friendly and do require some post processing in order to extract useful data.