Do "--pacing-timer" or "-b/burst_size" option work for TCP?

[sbeslagic]

Hi,

I'm seeing difference between TCP and UDP traffic when using "--pacing-timer" or "-b/burst_size" options. While these options work for UDP as I expect them to and packets get more evenly spaced, the TCP traffic seems to consist of a full 10GE burst, then waiting, then another 10GE burst, ... and so on. UDP with "-b/1" gets me the most evenly spaced frames, while again it does nothing for the TCP.

I'm using iperf3 3.16 with 10GE Mellanox ConnectX-4 Lx NIC on a Ubuntu PC.
Traffic was captured with an optical TAP feeding another 10GE NIC, connected right after iperf's client 10GE port.

ip netns exec uplink iperf3 -c 192.168.99.3 -b 10M --pacing-timer 1000

Connecting to host 192.168.99.3, port 5201
[ 5] local 192.168.99.1 port 48830 connected to 192.168.99.3 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.25 MBytes 10.5 Mbits/sec 0 134 KBytes
[ 5] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec 0 134 KBytes
[ 5] 2.00-3.00 sec 1.12 MBytes 9.44 Mbits/sec 0 134 KBytes
[ 5] 3.00-4.00 sec 1.25 MBytes 10.5 Mbits/sec 0 134 KBytes
[ 5] 4.00-5.00 sec 1.12 MBytes 9.44 Mbits/sec 0 134 KBytes
[ 5] 5.00-6.00 sec 1.25 MBytes 10.5 Mbits/sec 0 134 KBytes
[ 5] 6.00-7.00 sec 1.12 MBytes 9.44 Mbits/sec 0 134 KBytes
[ 5] 7.00-8.00 sec 1.25 MBytes 10.5 Mbits/sec 0 134 KBytes
[ 5] 8.00-9.00 sec 1.12 MBytes 9.44 Mbits/sec 0 134 KBytes
[ 5] 9.00-10.00 sec 1.25 MBytes 10.5 Mbits/sec 0 134 KBytes

---

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 12.0 MBytes 10.1 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 12.0 MBytes 10.1 Mbits/sec receiver

tshark with "-T fields -e frame.time_delta"

0.000001216
0.000001223
0.000001235
0.000001229
0.000001228
0.000022036
0.104721280
0.000001177
0.000001235
0.000001229
0.000001242
0.000001216
0.000001293

ip netns exec uplink iperf3 -c 192.168.99.3 -b 10M --pacing-timer 1000 -u

Connecting to host 192.168.99.3, port 5201
[ 5] local 192.168.99.1 port 44794 connected to 192.168.99.3 port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 1.19 MBytes 10.0 Mbits/sec 863
[ 5] 1.00-2.00 sec 1.19 MBytes 10.0 Mbits/sec 863
[ 5] 2.00-3.00 sec 1.19 MBytes 10.0 Mbits/sec 863
[ 5] 3.00-4.00 sec 1.19 MBytes 10.0 Mbits/sec 864
[ 5] 4.00-5.00 sec 1.19 MBytes 10.0 Mbits/sec 863
[ 5] 5.00-6.00 sec 1.19 MBytes 10.0 Mbits/sec 863
[ 5] 6.00-7.00 sec 1.19 MBytes 10.0 Mbits/sec 864
[ 5] 7.00-8.00 sec 1.19 MBytes 10.0 Mbits/sec 863
[ 5] 8.00-9.00 sec 1.19 MBytes 10.0 Mbits/sec 863
[ 5] 9.00-10.00 sec 1.19 MBytes 10.0 Mbits/sec 863

---

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 11.9 MBytes 10.0 Mbits/sec 0.000 ms 0/8632 (0%) sender
[ 5] 0.00-10.00 sec 11.9 MBytes 10.0 Mbits/sec 0.006 ms 0/8632 (0%) receiver

tshark with "-T fields -e frame.time_delta"

0.001000224
0.000950534
0.001050131
0.000999450
0.002000186
0.000999628
0.001000039
0.000999891
0.000950624
0.001049549
0.001999993
0.000950605
0.001049197
0.000999533

[davidBar-On]

Hi, note that -b and --pacing-timer are actually mutual exclusive, and when both are defined, only the -b is used. I.e. the tests are set for 10Mbps.

There are two differences between the UDP and TCP tests that may help to explain the results:

1. UDP: default packet size is usually 1460 bytes (use -V to see the actual value in your tests). For 10Mbps that means about 900 packets per second, i.e. a packet every 0.0011 sec, which about what is seen in the tshark output.
2. TCP: default packet size is 128KB, i.e. 10 packets for 10Mbps - a packet each 0.1 sec. Also, TCP is a streaming protocol - in general the packets or segments sent over the network are not the send() packets. Therefor, for each 128KB sent by iperf3 several packets are sent over the network. The result is that for each 128KB iperf3 sends every 0.1 sec, for a short time several packets are sent (about every 0.0000012 sec per tshark output) until all the 128KB are sent. Then there is a long wait until the next 128KB (this is probably the 0.104721280 delta time). To have the same packet size as in UDP, use -l 1460. Note however that since TCP is a streaming protocol and also add ACK messages, the result may not be the same as for UDP.

[sbeslagic]

Hi,

thank you very much for your reply, I was not aware of the TCP default packet size value, I was just looking at Ethernet level. It all makes sense now. The "-l" option does have an effect on the TCP that I was looking for.