Simula Research Laboratory / Center for Resilient Networks and Applications / NorNet
Homepage of Thomas Dreibholz / NetPerfMeter Homepage

NetPerfMeter
A TCP/MPTCP/UDP/SCTP/DCCP Network Performance Meter Tool
NetPerfMeter Logo

📑 Quick Navigation

📰 Latest News

💡 What is Network Performance Meter (NetPerfMeter)?

NetPerfMeter is a network performance meter for the TCP, MPTCP, SCTP, UDP, and DCCP transport protocols over IPv4 and IPv6. It simultaneously transmits bidirectional flows to an endpoint and measures the resulting flow bandwidths and QoS. Flows can be saturated (i.e. “send as much as possible”) or non-saturated with frame rate and frame sizes (like a multimedia transmission). Non-saturated flows can be configured with constant or variable frame rate/frame size, i.e. to realise Constant Bit Rate (CBR) or Variable Bit Rate (VBR) traffic. For both, frame rate and frame size, it is not only possible to set constant values but to also to use random distributions. Furthermore, flows can be set up as on/off flows. Of course, the flow parameters can be configured individually per flow and flow direction. The measurement results can be recorded as scalar files (summary of the run) and vector files (time series). These files can be processed further, e.g. for detailed analysis and plotting of the results. The Wireshark network protocol analyser provides out-of-the-box support for analysing NetPerfMeter packet traffic.

Screenshot of NetPerfMeter run
A NetPerfMeter Run with an SCTP Flow

Design Goals and Features

The key goal of NetPerfMeter is to provide a tool for the performance comparison of multiple transport connections, which are further denoted as Flows. That is, it is possible to configure different flows between two systems using varying parameters, in order run a configured measurement, collect the obtained results and post-process them for statistical analyses. Particularly, all five relevant IETF Transport Layer protocols are supported:

Of course, this support includes the possibility to parametrise various protocol-specific options. Note, that the protocol support by NetPerfMeter depends on the underlying operating system. DCCP, MPTCP, as well as some SCTP extensions are not available on all platforms, yet.

Furthermore, each flow is able to apply its specific traffic behaviour:

Clearly, the NetPerfMeter application provides features similar to the NetPerfMeter simulation model in OMNeT++. It is therefore relatively easy – from the parametrisation perspective – to reproduce NetPerfMeter simulation scenarios in reality.

Instances and Protocols

Figure of the Concept of a NetPerfMeter Measurement
The Concept of a NetPerfMeter Measurement

Similar to the NetPerfMeter simulation model in OMNeT++, an application instance may either be in Active Mode (client side) or Passive Mode (server side). The figure above illustrates the concept of a NetPerfMeter measurement. The passive instance accepts incoming NetPerfMeter connections from the active instance. The active instance controls the passive instance, by using a control protocol denoted as NetPerfMeter Control Protocol (NPMP-CONTROL). That is, the passive instance may run as a daemon; no manual interaction by the user – e.g. to restart it before a new measurement run – is required. This feature is highly practical for a setup distributed over multiple Internet sites (e.g. like the NorNet Testbed) and allows for parameter studies consisting of many measurement runs.

The payload data between active and passive instances is transported using the NetPerfMeter Data Protocol (NPMP-DATA). The figure below shows the protocol stack of a NetPerfMeter node.

Figure of the NetPerfMeter Protocol Stack
The NetPerfMeter Protocol Stack

The NPMP-DATA protocol transmits data as frames, with a given frame rate. In case of a saturated sender, the flow tries to send as many frames as possible (i.e. as allowed by the underlying transport and its flow and congestion control). Otherwise, the configured frame rate is used (e.g. 25 frames/s as for typical video transmissions). NPMP-DATA breaks down frames into messages, to make sure that large frames can be transported over the underlying transport protocol. The maximum message size can be configured. Frames larger than the message size limit are split into multiple messages before sending them. On the receiving side, the messages are combined back into frames. The underlying transport protocol handles the messages as its payload.

Measurement Processing

The following figure presents the message sequence of a NetPerfMeter measurement run:

Figure of a Measurement Run with NetPerfMeter
A Measurement Run with NetPerfMeter

Note that the Wireshark network protocol analyser provides out-of-the-box support for NetPerfMeter. That is, it is able to dissect and further analyse NPMP-CONTROL and NPMP-DATA packets using all supported Transport Layer protocols.

Measurement Setup

A new measurement run setup is initiated by the active NetPerfMeter instance by establishing an NPMP-CONTROL association to the passive instance first. The NPMP-CONTROL association by default uses SCTP for transport. If SCTP is not possible in the underlying networks (e.g. due to firewalling restrictions), it is optionally possible to use TCP for the NPMP-CONTROL association instead. Then, the configured NPMP-DATA connections are established by their configured Transport Layer protocols. For the connection-less UDP, the message transfer is just started. The passive NetPerfMeter instance is informed about the identification and parameters of each new flow by using NPMP-CONTROL Add Flow messages. On startup of the NPMP-DATA flow, an NPMP-DATA Identify message allows the mapping of a newly incoming connection to a configured flow by the passive instance. It acknowledges each newly set up flow by an NPMP-CONTROL Acknowledge message. After setting up all flows, the scenario is ready to start the measurement run.

Measurement Run

The actual measurement run is initiated from the active NetPerfMeter instance using an NPMP-CONTROL Start Measurement message, which is also acknowledged by an NPMP-CONTROL Acknowledge message. Then, both instances start running the configured scenario by transmitting NPMP-DATA Data messages over their configured flows.

During the measurement run, incoming and outgoing flow bandwidths may be recorded as vectors – i.e. time series – at both instances, since NPMP-DATA Data traffic may be bidirectional. Furthermore, the CPU utilisations – separately for each CPU and CPU core – are also tracked. This allows to identify performance bottlenecks, which is particularly useful when debugging and comparing transport protocol implementation performance. Furthermore, the one-way delay of messages can be recorded. Of course, in order to use this feature, the clocks of both nodes need to be appropriately synchronised, e.g. by using the Network Time Protocol (NTP).

Measurement Termination

The end of a measurement run is initiated – from the active NetPerfMeter instance – by using an NPMP-CONTROL Stop Measurement message. Again, it is acknowledged by an NPMP-CONTROL Acknowledge message. At the end of the measurement, average bandwidth and one-way delay of each flow and stream are recorded as scalars (i.e. single values). They may provide an overview of the long-term system performance.

Result Collection

After stopping the measurement, the passive NetPerfMeter instance sends its global vector and scalar results (i.e. over all flows) to the active instance, by using one or more NPMP-CONTROL Results messages. Then, the active NetPerfMeter instance sequentially removes the flows by using NPMP-CONTROL Remove Flow messages, which are acknowledged by NPMP-CONTROL Acknowledge messages. On flow removal, the passive instance sends its per-flow results for the corresponding flow, again by using NPMP-CONTROL Results messages.

The active instance, as well, archives its local vector and scalar results data and stores them – together with the results received from its peer – locally. All result data is compressed by using BZip2 compression (see bzip2), which may save a significant amount of bandwidth (of course, the passive node compresses the data before transfer) and disk space.

Measurement Execution, Result Post-Processing and Visualisation

By using shell scripts, it is possible to apply NetPerfMeter for parameter studies, i.e. to create a set of runs for each input parameter combination. For example, a script could iterate over a send buffer size σ from 64 KiB to 192 KiB in steps of 64 KiB as well as a path bandwidth ρ from 10 Mbit/s to 100 Mbit/s in steps of 10 Mbit/s and perform 5 measurement runs for each parameter combination.

When all measurement runs have eventually been processed, the results have to be visualised for analysis and interpretation. The NetPerfMeter package provides support to visualise the scalar results, which are distributed over the scalar files written by measurement runs. Therefore, the first step necessary is to bring the data from the various scalar files into an appropriate form for further post-processing. This step is denoted as Summarisation; an introduction is also provided in “SimProcTC – The Design and Realization of a Powerful Tool-Chain for OMNeT++ Simulations”.

The summarisation task is performed by the tool createsummary. An external program – instead of just using GNU R itself to perform this step – is used due to the requirements on memory and CPU power. createsummary iterates over all scalar files of a measurement M. Each file is read – with on-the-fly BZip2-decompression – and each scalar value as well as the configuration m∈M having led to this value – are stored in memory. Depending on the number of scalars, the required storage space may have a size of multiple GiB.

Since usually not all scalars of a measurement are required for analysis (e.g. for an SCTP measurement, it may be unnecessary to include unrelated statistics), a list of scalar name prefixes to be excluded from summarisation can be provided to createsummary, in form of the so-called Summary Skip List. This feature may significantly reduce the memory and disk space requirements of the summarisation step. Since the skipped scalars still remain stored in the scalar files themselves, it is possible to simply re-run createsummary with updated summary skip list later, in order to also include them.

Having all relevant scalars stored in memory, a data file – which can be processed by GNU R, LibreOffice or other programs – is written for each scalar. The data file is simply a table in text form, containing the column names on the first line. Each following line contains the data, with line number and an entry for each column (all separated by spaces); an example is provided in Listing 3 of “SimProcTC – The Design and Realization of a Powerful Tool-Chain for OMNeT++ Simulations”. That is, each line consists of the settings of all parameters and the resulting scalar value. The data files are also BZip2-compressed on the fly, in order to reduce the storage space requirements.

😀 Examples

Preparations

NetPerfMeter by default uses the SCTP protocol for control communication (which can be changed by the -control-over-tcp option, see later). It may be useful to allow loading the SCTP kernel module first, if not already enabled. The following code blocks show how to enable it permanently.

SCTP on Linux

echo "sctp" | sudo tee /etc/modules-load.d/sctp.conf
if [ -e /etc/modprobe.d/sctp-blacklist.conf ] ; then
   sudo sed -e 's/^blacklist sctp/# blacklist sctp/g' -i /etc/modprobe.d/sctp-blacklist.conf
fi
sudo modprobe sctp
lsmod | grep sctp

SCTP on FreeBSD

echo 'sctp_load="YES"' | sudo tee --append /boot/loader.conf
sudo kldload sctp
kldstat | grep sctp

Starting the Passive Instance (Server)

Running the Active Instance (Client)

Simple TCP Communication

Simple Non-TCP Communication

Variable Bitrate Flows

NetPerfMeter supports randomised frame rate and frame size, to create variable bitrate (VBR) flows. The following distributions are currently available:

A configured distribution is used to determine:

Some examples:

Multiple Flows and Measurement Results Recording

Wireshark

Screenshot of NetPerfMeter run
A Wireshark Run with NetPerfMeter Traffic from multi.pcap.gz

Miscellaneous

Running Larger-Scale Measurements using the CreateSummary and CombineSummaries Tools

Goal of many NetPerfMeter measurements is likely to perform larger-scale measurements, with multiple NetPerfMeter runs for every combination of NetPerfMeter as well as non-NetPerfMeter parameters.

Creating a Run Script

Example “experiment1”:

The corresponding measurement can be implemented as script (in arbitrary language, e.g. as a simple shell script), basically implementing something like this: 

#!/bin/sh -eu

NAME="experiment1"
RUNTIME=60
DESTINATIONS="10.10.10.10 172.20.30.40 fd91:b3aa:b9c:beef::10 fdd8:c818:a429:cafe:40"
FLOWS="const0:const1024:const0:const1024"
PROTOCOLS="tcp sctp"
OPTION1="value1 value2 value3"
OPTION2="test1 test2 test3"

# ------ Prepare results directory --------------------------------------------------------
mkdir -p "$NAME"
cd "$NAME"

for destination in $DESTINATIONS ; do
   for flow in $FLOWS ; do
      for protocol in $PROTOCOLS ; do
         for option1 in $OPTION1 ; do
            for option2 in $OPTION2 ; do

               # ------ Prepare a unique directory for the results ------------------------
               now="$(date -u -Iseconds)"   # <- Unique name by using date
               run="$now:$destination-$flow-$protocol-$option1-$option2"
               directory="run-$(echo "$run" | sha1sum | cut -d' ' -f1)"
               mkdir -p "$directory"

               # ------ Prepare name/patterns for results files ---------------------------
               scalarPattern="$directory/run.sca.bz2"
               vectorPattern="$directory/run.vec.bz2"
               configName="$directory/run.config"

               # Do something, to configure non-NetPerfMeter options
               something-to-configure-option1 "$option1"
               something-to-configure-option2 "$option2"
               ...

               # ------ Run NetPerfMeter --------------------------------------------------
               # NOTE: Add -vector="$vectorPattern", if needed
               netperfmeter "[$destination]:9000" \
                  -scalar="$scalarPattern" \
                  -config="$configName" \
                  -runtime=$RUNTIME \
                  "-$protocol" const0:const1024:const0:const1024

            done
         done
      done
   done
done

Notes:

The result of the script execution is a directory experiment1, with one subdirectory run-<HASH> for each NetPerfMeter run. Each of these subdirectories will contain the scalar files run-active.sca.bz2 (active-side results) and run-passive.sca.bz2 (passive-side results), with all written scalars.

Applying CreateSummary

Clearly, the goal is to create a summary for each scalar, i.e. a table with columns for each parameter setting and the resulting scalar value, i.e. for the scalar passive.flow-ReceivedBitRate:

Destination Protocol Option1 Option2 Directory passive.flow-ReceivedBitRate
10.10.10.10 tcp value1 test1 run-<HASH1> 12345678
10.10.10.10 tcp value1 test2 run-<HASH2> 11111111
… … … … run-<HASH…> …
fdd8:c818:a429:cafe:40 sctp value3 test3 run-<HASH_LAST> 11223344

The summarisation task can be realised by the tool CreateSummary. It generates the output tables (BZip2-compressed CSV format, using TAB as delimiter) from all the scalar files of the measurement. For this summarisation, it needs information about:

In the example above, this information needs to be added by preparing an input file results.summary, and then process this input by CreateSummary: 

# ------ Prepare results directory --------------------------------------------------------
...
if [ ! -e results.summary ] ; then
   echo "--varnames=Destination Protocol Option1 Option2 Directory" >results.summary
fi

for destination in $DESTINATIONS ; do
   ...
            ...

               # ------ Run NetPerfMeter --------------------------------------------------
               ...

               # ------ Append run to results.summary -------------------------------------
               (
                  echo "--values=$destination $flow $protocol $option1 $option2 $directory"
                  echo "--input=$directory/run-active.sca.bz2"
                  echo "--values=$destination $flow $protocol $option1 $option2 $directory"
                  echo "--input=$directory/run-passive.sca.bz2"
               ) >>results.summary

            ...
   ...
done

# ------ Run CreateSummary ----------------------------------------------------------------
createsummary -batch <results.summary

The full script is available in: run-experiment1.

Notes:

The result of the script execution is a BZip2-compressed CSV table for each scalar, e.g. active.flow-ReceivedBitRate.data.bz2 and passive.flow-ReceivedBitRate.data.bz2 containing the received bit rates of active and passive side. These files can be loaded into arbitrary tools handling CSV files. If necessary TAB needs to be specified as delimiter. For example, in GNU R: 

library("data.table")

results <- fread("experiment1/active.flow-ReceivedBitRate.data.bz2")

print(summary(results))                                    # Show data summary
print(colnames(results))                                   # Show table columns
print(results$"active.flow-ReceivedBitRate" / 1000000.0)   # Received bit rate in Mbit/s

Applying CombineSummaries

In some cases, it may be necessary to combine summary tables written by CreateSummary. For example, measurements have been from hosts host1.example and host2.example, now having collected data from both hosts. For analysis, the results in separate files (i.e. tables) for each host can be combined into a single file, with a new table column “Host” containing the measurement host: 

(
  echo '--values="host1.example"'
  echo '--input=host1/experiment1/active.flow-ReceivedBitRate.data.bz2'
  echo '--values="host2.example"'
  echo '--input=host2/experiment1/active.flow-ReceivedBitRate.data.bz2'
) | combinesummaries combined-active.flow-ReceivedBitRate.data.bz2 "Host"

Then, after loading the resulting combined file combined-active.flow-ReceivedBitRate.data.bz2 into an analysis tool like GNU R, the information about the host is in the added table column “Host”.

📦 Binary Package Installation

Please use the issue tracker at https://github.com/dreibh/netperfmeter/issues to report bugs and issues!

Ubuntu Linux

For ready-to-install Ubuntu Linux packages of NetPerfMeter, see Launchpad PPA for Thomas Dreibholz! 

sudo apt-add-repository -sy ppa:dreibh/ppa
sudo apt-get update
sudo apt-get install netperfmeter-all

Fedora Linux

For ready-to-install Fedora Linux packages of NetPerfMeter, see COPR PPA for Thomas Dreibholz! 

sudo dnf copr enable -y dreibh/ppa
sudo dnf install netperfmeter-all

FreeBSD

For ready-to-install FreeBSD packages of NetPerfMeter, it is included in the ports collection, see FreeBSD ports tree index of benchmarks/netperfmeter/! 

sudo pkg install netperfmeter

Alternatively, to compile it from the ports sources: 

cd /usr/ports/benchmarks/netperfmeter
make
sudo make install

💾 Build from Sources

NetPerfMeter is released under the GNU General Public Licence (GPL).

Please use the issue tracker at https://github.com/dreibh/netperfmeter/issues to report bugs and issues!

Development Version

The Git repository of the NetPerfMeter sources can be found at https://github.com/dreibh/netperfmeter: 

git clone https://github.com/dreibh/netperfmeter
cd netperfmeter
sudo ci/get-dependencies --install
cmake .
make

Note: The script ci/get-dependencies automatically installs the build dependencies under Debian/Ubuntu Linux, Fedora Linux, and FreeBSD. For manual handling of the build dependencies, see the packaging configuration in debian/control (Debian/Ubuntu Linux), netperfmeter.spec (Fedora Linux), and Makefile FreeBSD.

Contributions:

Current Stable Release

The tarball has been signed with my GnuPG key 21412672­518D8B2D­1862EFEF­5CD5D12A­A0877B49. Its authenticity and integrity can be verified by: 

gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys 21412672518D8B2D1862EFEF5CD5D12AA0877B49
gpg --verify netperfmeter-<VERSION>.tar.xz.asc netperfmeter-<VERSION>.tar.xz

Old Stable Releases

The tarballs have been signed with my GnuPG key 21412672­518D8B2D­1862EFEF­5CD5D12A­A0877B49. Its authenticity and integrity can be verified by: 

gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys 21412672518D8B2D1862EFEF5CD5D12AA0877B49
gpg --verify netperfmeter-<VERSION>.tar.xz.asc netperfmeter-<VERSION>.tar.xz

🖋️ Citing NetPerfMeter in Publications

NetPerfMeter BibTeX entries can be found in netperfmeter.bib!

📖 Publications

The collected BibTeX references in a single file are available as: BiBTeX file, XML file, ODT file, DOCX file. These lists were generated using BibTeXCov 2.0!

2025

2022

2015

2012

2011

2010

2009

2008

©️ 01.12.2025 Thomas Dreibholz, Simula Research Laboratory / Center for Resilient Networks and Applications / NorNet

Valid XHTML 1.1! Valid CSS!