RW Daily Logs

1st December 2005

11:11

We don't know if the problem also appears in the 2.4 kernel: 2.4 performances are far above performances on 2.6, which could be a sufficient factor to not trigger to problem.

21th November 2005

18:16

Pathrate (http://www.cc.gatech.edu/fac/Constantinos.Dovrolis/pathrate.html), "A measurement tool for the capacity of network paths" seems to have problems has well to establish a satisfactory TCP connection, while it is working just fine on the lan. It seems to wait forever after measuring the RTT.

17:08

An extra comment: the iperf server is reporting what is seen in TCPDump.

A bad session on the client:

# iperf -c gridui01.usatlas.bnl.gov -p21212 -t 12000 -i 2 -w 8M
------------------------------------------------------------
Client connecting to gridui01.usatlas.bnl.gov, TCP port 21212
TCP window size: 16.0 MByte (WARNING: requested 8.00 MByte)
------------------------------------------------------------
[  3] local 128.142.224.234 port 32778 connected with 130.199.185.10 port 21212
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 2.0 sec  11.9 MBytes  49.9 Mbits/sec
[  3]  2.0- 4.0 sec  12.5 MBytes  52.6 Mbits/sec
[  3]  4.0- 6.0 sec  10.9 MBytes  45.6 Mbits/sec
[  3]  6.0- 8.0 sec  0.00 Bytes  0.00 bits/sec
[  3]  8.0-10.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 10.0-12.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 12.0-14.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 14.0-16.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 16.0-18.0 sec  0.00 Bytes  0.00 bits/sec

On the iperf server, we could see:

[rwartel@gridui01 ~]$ iperf -s -p 21212 -w 8M -i 2
------------------------------------------------------------
Server listening on TCP port 21212
TCP window size: 16.0 MByte (WARNING: requested 8.00 MByte)
------------------------------------------------------------
[  4] local 130.199.185.10 port 21212 connected with 128.142.224.234 port 32778
[  4]  0.0- 2.0 sec  2.53 MBytes  10.6 Mbits/sec
[  4]  2.0- 4.0 sec  8.23 MBytes  34.5 Mbits/sec
[  4]  4.0- 6.0 sec  9.51 MBytes  39.9 Mbits/sec
[  4]  6.0- 8.0 sec  1.23 MBytes  5.15 Mbits/sec
[  4]  8.0-10.0 sec  6.68 KBytes  27.4 Kbits/sec
[  4] 10.0-12.0 sec  5.34 KBytes  21.9 Kbits/sec
[  4] 12.0-14.0 sec  6.68 KBytes  27.4 Kbits/sec
[  4] 14.0-16.0 sec  5.34 KBytes  21.9 Kbits/sec
[  4] 16.0-18.0 sec  6.68 KBytes  27.4 Kbits/sec

11th November 2005

13:54

Network analysis of a typical iperf session:

[root@lxshare220d ~]# iperf -c rembrandt3.grid.sara.nl  -t 12000 -i 2 -w 2M -p50 01
------------------------------------------------------------
Client connecting to rembrandt3.grid.sara.nl, TCP port 5001
TCP window size: 4.00 MByte (WARNING: requested 2.00 MByte)
------------------------------------------------------------
[  3] local 128.142.224.220 port 32772 connected with 145.100.3.85 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 2.0 sec   143 MBytes   600 Mbits/sec
[  3]  2.0- 4.0 sec   166 MBytes   697 Mbits/sec
[  3]  4.0- 6.0 sec   170 MBytes   712 Mbits/sec
[  3]  6.0- 8.0 sec   170 MBytes   712 Mbits/sec
[  3]  8.0-10.0 sec   168 MBytes   704 Mbits/sec
[  3] 10.0-12.0 sec   171 MBytes   718 Mbits/sec
[  3] 12.0-14.0 sec   172 MBytes   722 Mbits/sec
[  3] 14.0-16.0 sec   171 MBytes   718 Mbits/sec
[  3] 16.0-18.0 sec   174 MBytes   730 Mbits/sec
[  3] 18.0-20.0 sec   176 MBytes   736 Mbits/sec
[  3] 20.0-22.0 sec   176 MBytes   737 Mbits/sec
[  3] 22.0-24.0 sec   171 MBytes   717 Mbits/sec
[  3] 24.0-26.0 sec   134 MBytes   561 Mbits/sec
[  3] 26.0-28.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 28.0-30.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 30.0-32.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 32.0-34.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 34.0-36.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 36.0-38.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 38.0-40.0 sec  0.00 Bytes  0.00 bits/sec
[ ID] Interval       Transfer     Bandwidth
[  3] 40.0-42.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 42.0-44.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 44.0-46.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 46.0-48.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 48.0-50.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 50.0-52.0 sec  0.00 Bytes  0.00 bits/sec
<SNIP still nothing at all>
[ ID] Interval       Transfer     Bandwidth
[  3] 200.0-202.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 202.0-204.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 204.0-206.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 206.0-208.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 208.0-210.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 210.0-212.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 212.0-214.0 sec   127 MBytes   532 Mbits/sec
[  3] 214.0-216.0 sec   169 MBytes   709 Mbits/sec
[  3] 216.0-218.0 sec   170 MBytes   712 Mbits/sec
[  3] 218.0-220.0 sec   167 MBytes   701 Mbits/sec
[  3] 220.0-222.0 sec   168 MBytes   706 Mbits/sec
[  3] 222.0-224.0 sec   162 MBytes   678 Mbits/sec
[  3] 224.0-226.0 sec   165 MBytes   693 Mbits/sec
[  3] 226.0-228.0 sec   164 MBytes   689 Mbits/sec
[  3] 228.0-230.0 sec   170 MBytes   712 Mbits/sec
[  3] 230.0-232.0 sec   167 MBytes   701 Mbits/sec
[  3] 232.0-234.0 sec   167 MBytes   699 Mbits/sec
[  3] 234.0-236.0 sec   165 MBytes   693 Mbits/sec
[  3] 236.0-238.0 sec   167 MBytes   699 Mbits/sec
[  3] 238.0-240.0 sec   160 MBytes   669 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3] 240.0-242.0 sec   166 MBytes   694 Mbits/sec
[  3] 242.0-244.0 sec   167 MBytes   700 Mbits/sec
[  3] 244.0-246.0 sec   167 MBytes   699 Mbits/sec
<SNIP throughput stable and acceptable>

TCPDUMP with comments based on iperf reports:

# iperf starts OK:
13:47:41.985081 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . ack 1 win 1460 <nop,nop,timestamp 892183 680456365>
13:47:41.985626 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1:25(24) ack 1 win 1460 <nop,nop,timestamp 892184 680456365>
13:47:41.985887 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 25:1473(1448) ack 1 win 1460 <nop,nop,timestamp 892184 680456365>
13:47:41.985923 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1473:2921(1448) ack 1 win 1460 <nop,nop,timestamp 892184 680456365>
13:47:42.008554 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 25 win 23 <nop,nop,timestamp 680456389 892184>
13:47:42.008607 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2921:4369(1448) ack 1 win 1460 <nop,nop,timestamp 892207 680456389>
13:47:42.008615 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 4369:5817(1448) ack 1 win 1460 <nop,nop,timestamp 892207 680456389>
13:47:42.008797 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1473 win 34 <nop,nop,timestamp 680456389 892184>
13:47:42.008806 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 5817:7265(1448) ack 1 win 1460 <nop,nop,timestamp 892207 680456389>
13:47:42.008811 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 7265:8713(1448) ack 1 win 1460 <nop,nop,timestamp 892207 680456389>
13:47:42.008815 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2921 win 46 <nop,nop,timestamp 680456389 892184>
13:47:42.008821 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 8713:10161(1448) ack 1 win 1460 <nop,nop,timestamp 892207 680456389>
13:47:42.008826 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 10161:11609(1448) ack 1 win 1460 <nop,nop,timestamp 892207 680456389>
13:47:42.031657 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 4369 win 57 <nop,nop,timestamp 680456412 892207>
13:47:42.031665 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 11609:13057(1448) ack 1 win 1460 <nop,nop,timestamp 892230 680456412>
13:47:42.031670 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 13057:14505(1448) ack 1 win 1460 <nop,nop,timestamp 892230 680456412>
13:47:42.031674 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 5817 win 68 <nop,nop,timestamp 680456412 892207>
13:47:42.031680 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 14505:15953(1448) ack 1 win 1460 <nop,nop,timestamp 892230 680456412>
13:47:42.031684 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 15953:17401(1448) ack 1 win 1460 <nop,nop,timestamp 892230 680456412>
13:47:42.031687 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 7265 win 80 <nop,nop,timestamp 680456412 892207>
13:47:42.031693 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 17401:18849(1448) ack 1 win 1460 <nop,nop,timestamp 892230 680456412>
13:47:42.031698 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 18849:20297(1448) ack 1 win 1460 <nop,nop,timestamp 892230 680456412>
13:47:42.031701 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 8713 win 91 <nop,nop,timestamp 680456412 892207>
<SNIP>
# iperf is not reporting any problem:
13:47:42.123757 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 305553:307001(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456504>
13:47:42.123761 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 152065 win 1211 <nop,nop,timestamp 680456504 892299>
13:47:42.123766 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 307001:308449(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456504>
13:47:42.123770 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 308449:309897(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456504>
13:47:42.123773 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 153513 win 1222 <nop,nop,timestamp 680456504 892299>
13:47:42.123779 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 309897:311345(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456504>
13:47:42.123783 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 311345:312793(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456504>
13:47:42.123785 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 154961 win 1234 <nop,nop,timestamp 680456504 892299>
13:47:42.123791 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 312793:314241(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456504>
13:47:42.123795 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 314241:315689(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456504>
13:47:42.123857 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 156409 win 1245 <nop,nop,timestamp 680456504 892299>
13:47:42.123864 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 315689:317137(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456504>
13:47:42.123869 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 317137:318585(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456504>
13:47:42.123871 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 157857 win 1256 <nop,nop,timestamp 680456504 892299>
13:47:42.123877 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 318585:320033(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456504>
13:47:42.123881 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 320033:321481(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456504>
13:47:42.123885 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 159305 win 1267 <nop,nop,timestamp 680456504 892299>
13:47:42.123890 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 321481:322929(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456504>
13:47:42.123894 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 322929:324377(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456504>
13:47:42.123897 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 160753 win 1279 <nop,nop,timestamp 680456504 892299>
13:47:42.123903 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 324377:325825(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456504>
13:47:42.123907 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 325825:327273(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456504>
13:47:42.123910 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 162201 win 1290 <nop,nop,timestamp 680456504 892299>
13:47:42.123915 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 327273:328721(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456504>
13:47:42.123919 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 328721:330169(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456504>
13:47:42.123922 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 163649 win 1301 <nop,nop,timestamp 680456504 892299>
13:47:42.123928 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 330169:331617(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456504>
13:47:42.123932 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 331617:333065(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456504>
13:47:42.123935 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 165097 win 1313 <nop,nop,timestamp 680456504 892299>
13:47:42.123940 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 333065:334513(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456504>
13:47:42.123944 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 334513:335961(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456504>
13:47:42.123947 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 166545 win 1324 <nop,nop,timestamp 680456504 892299>
13:47:42.124111 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 170889 win 1358 <nop,nop,timestamp 680456505 892299>
13:47:42.124119 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 344649:346097(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456505>
13:47:42.124123 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 346097:347545(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456505>
13:47:42.124126 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 172337 win 1369 <nop,nop,timestamp 680456505 892299>
13:47:42.124132 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 347545:348993(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456505>
13:47:42.124136 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 348993:350441(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456505>
13:47:42.124139 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 173785 win 1381 <nop,nop,timestamp 680456505 892299>
13:47:42.124147 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 350441:351889(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456505>
13:47:42.124152 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 351889:353337(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456505>
13:47:42.124155 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 175233 win 1392 <nop,nop,timestamp 680456505 892299>
13:47:42.124160 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 353337:354785(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456505>
13:47:42.124165 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 354785:356233(1448) ack 1 win 1460 <nop,nop,timestamp 892322 680456505>
13:47:42.124495 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 205641 win 1629 <nop,nop,timestamp 680456505 892299>
13:47:42.124504 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 414153:415601(1448) ack 1 win 1460 <nop,nop,timestamp 892323 680456505>
13:47:42.124508 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 415601:417049(1448) ack 1 win 1460 <nop,nop,timestamp 892323 680456505>
13:47:42.124511 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 207089 win 1641 <nop,nop,timestamp 680456505 892299>
13:47:42.146837 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 273697 win 2161 <nop,nop,timestamp 680456527 892322>
13:47:42.146847 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 550265:551713(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146852 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 551713:553161(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146855 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 275145 win 2172 <nop,nop,timestamp 680456527 892322>
13:47:42.146861 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 553161:554609(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146865 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 554609:556057(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146868 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 276593 win 2184 <nop,nop,timestamp 680456527 892322>
13:47:42.146874 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 556057:557505(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146878 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 557505:558953(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146881 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 278041 win 2195 <nop,nop,timestamp 680456527 892322>
13:47:42.146887 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 558953:560401(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146891 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 560401:561849(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146894 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 279489 win 2206 <nop,nop,timestamp 680456527 892322>
13:47:42.146899 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 561849:563297(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146903 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 563297:564745(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146905 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 280937 win 2218 <nop,nop,timestamp 680456527 892322>
13:47:42.146911 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 564745:566193(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146915 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 566193:567641(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146917 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 282385 win 2229 <nop,nop,timestamp 680456527 892322>
13:47:42.146927 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 567641:569089(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146931 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 569089:570537(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146934 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 283833 win 2240 <nop,nop,timestamp 680456527 892322>
13:47:42.146940 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 570537:571985(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146944 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 571985:573433(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146947 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 285281 win 2252 <nop,nop,timestamp 680456527 892322>
13:47:42.146952 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 573433:574881(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146956 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 574881:576329(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146959 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 286729 win 2263 <nop,nop,timestamp 680456527 892322>
13:47:42.146965 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 576329:577777(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146969 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 577777:579225(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146972 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 288177 win 2274 <nop,nop,timestamp 680456527 892322>
13:47:42.146978 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 579225:580673(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146982 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 580673:582121(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146985 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 289625 win 2286 <nop,nop,timestamp 680456527 892322>
13:47:42.146990 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 582121:583569(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146995 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 583569:585017(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.146997 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 291073 win 2297 <nop,nop,timestamp 680456527 892322>
13:47:42.147003 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 585017:586465(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.147007 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 586465:587913(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.147010 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 292521 win 2308 <nop,nop,timestamp 680456527 892322>
13:47:42.147015 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 587913:589361(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.147019 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 589361:590809(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.147022 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 293969 win 2320 <nop,nop,timestamp 680456527 892322>
13:47:42.147028 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 590809:592257(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.147032 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 592257:593705(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.147035 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 295417 win 2331 <nop,nop,timestamp 680456527 892322>
13:47:42.147040 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 593705:595153(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.147045 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 595153:596601(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.147047 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 296865 win 2342 <nop,nop,timestamp 680456527 892322>
13:47:42.147053 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 596601:598049(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.147057 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 598049:599497(1448) ack 1 win 1460 <nop,nop,timestamp 892345 680456527>
13:47:42.147060 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 298313 win 2353 <nop,nop,timestamp 680456527 892322>
13:47:42.147759 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 366369 win 2885 <nop,nop,timestamp 680456528 892322>
13:47:42.147767 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 721129:722577(1448) ack 1 win 1460 <nop,nop,timestamp 892346 680456528>
13:47:42.147769 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 722577:724025(1448) ack 1 win 1460 <nop,nop,timestamp 892346 680456528>
13:47:42.147771 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 724025:725473(1448) ack 1 win 1460 <nop,nop,timestamp 892346 680456528>
13:47:42.147773 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 725473:726921(1448) ack 1 win 1460 <nop,nop,timestamp 892346 680456528>
13:47:42.147775 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 726921:728369(1448) ack 1 win 1460 <nop,nop,timestamp 892346 680456528>
13:47:42.149593 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1039689:1041137(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149596 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1041137:1042585(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149597 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1042585:1044033(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149599 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1044033:1045481(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149601 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1045481:1046929(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149603 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1046929:1048377(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149605 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1048377:1049825(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149607 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1049825:1051273(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149609 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1051273:1052721(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149611 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1052721:1054169(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149718 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1054169:1055617(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149720 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1055617:1057065(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149722 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1057065:1058513(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149724 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1058513:1059961(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149726 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1059961:1061409(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149728 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1061409:1062857(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149730 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1062857:1064305(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149732 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1064305:1065753(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149734 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1065753:1067201(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149735 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1067201:1068649(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149846 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1068649:1070097(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149848 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1070097:1071545(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149850 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1071545:1072993(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149852 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1072993:1074441(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149854 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1074441:1075889(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149856 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1075889:1077337(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149858 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1077337:1078785(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149859 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1078785:1080233(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149861 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1080233:1081681(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149970 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1081681:1083129(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149973 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1083129:1084577(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149975 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1084577:1086025(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149976 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1086025:1087473(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.149978 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1087473:1088921(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.150093 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1097609:1099057(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.150096 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1099057:1100505(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.150098 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1100505:1101953(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.150099 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1101953:1103401(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.150101 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1103401:1104849(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.150103 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1104849:1106297(1448) ack 1 win 1460 <nop,nop,timestamp 892347 680456529>
13:47:42.169698 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 551713 win 4333 <nop,nop,timestamp 680456550 892345>
13:47:42.169708 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1106297:1107745(1448) ack 1 win 1460 <nop,nop,timestamp 892368 680456550>
13:47:42.169713 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1107745:1109193(1448) ack 1 win 1460 <nop,nop,timestamp 892368 680456550>
13:47:42.169716 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 553161 win 4344 <nop,nop,timestamp 680456550 892345>
13:47:42.169722 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1109193:1110641(1448) ack 1 win 1460 <nop,nop,timestamp 892368 680456550>
13:47:42.169727 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1110641:1112089(1448) ack 1 win 1460 <nop,nop,timestamp 892368 680456550>
13:47:42.169730 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 554609 win 4356 <nop,nop,timestamp 680456550 892345>
13:47:42.169735 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1112089:1113537(1448) ack 1 win 1460 <nop,nop,timestamp 892368 680456550>
13:47:42.169740 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1113537:1114985(1448) ack 1 win 1460 <nop,nop,timestamp 892368 680456550>
13:47:42.169825 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 556057 win 4367 <nop,nop,timestamp 680456550 892345>
13:47:42.169832 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1114985:1116433(1448) ack 1 win 1460 <nop,nop,timestamp 892368 680456550>
13:47:42.169837 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1116433:1117881(1448) ack 1 win 1460 <nop,nop,timestamp 892368 680456550>
13:47:42.169840 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 557505 win 4378 <nop,nop,timestamp 680456550 892345>
13:47:42.169846 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1117881:1119329(1448) ack 1 win 1460 <nop,nop,timestamp 892368 680456550>
13:47:42.169851 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1119329:1120777(1448) ack 1 win 1460 <nop,nop,timestamp 892368 680456550>
13:47:42.169853 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 558953 win 4390 <nop,nop,timestamp 680456550 892345>
13:47:42.169859 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1120777:1122225(1448) ack 1 win 1460 <nop,nop,timestamp 892368 680456550>
13:47:42.169863 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1122225:1123673(1448) ack 1 win 1460 <nop,nop,timestamp 892368 680456550>
13:47:42.169866 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 560401 win 4401 <nop,nop,timestamp 680456550 892345>
13:47:42.169871 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1123673:1125121(1448) ack 1 win 1460 <nop,nop,timestamp 892368 680456550>
13:47:42.169875 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1125121:1126569(1448) ack 1 win 1460 <nop,nop,timestamp 892368 680456550>
13:47:42.169878 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 561849 win 4412 <nop,nop,timestamp 680456550 892345>
13:47:42.169884 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1126569:1128017(1448) ack 1 win 1460 <nop,nop,timestamp 892368 680456550>
13:47:42.169888 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1128017:1129465(1448) ack 1 win 1460 <nop,nop,timestamp 892368 680456550>
13:47:42.169890 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 563297 win 4424 <nop,nop,timestamp 680456550 892345>
13:47:42.169896 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1129465:1130913(1448) ack 1 win 1460 <nop,nop,timestamp 892368 680456550>
13:47:42.169900 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1130913:1132361(1448) ack 1 win 1460 <nop,nop,timestamp 892368 680456550>
13:47:42.169953 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 564745 win 4435 <nop,nop,timestamp 680456550 892345>
13:47:42.169960 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1132361:1133809(1448) ack 1 win 1460 <nop,nop,timestamp 892368 680456550>
13:47:42.169965 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1133809:1135257(1448) ack 1 win 1460 <nop,nop,timestamp 892368 680456550>
13:47:42.169968 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 566193 win 4446 <nop,nop,timestamp 680456550 892345>
13:47:42.169974 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1135257:1136705(1448) ack 1 win 1460 <nop,nop,timestamp 892368 680456550>
13:47:42.169978 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1136705:1138153(1448) ack 1 win 1460 <nop,nop,timestamp 892368 680456550>
<SNIP>
13:47:42.171353 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1313361:1314809(1448) ack 1 win 1460 <nop,nop,timestamp 892370 680456552>
13:47:42.171356 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 655969 win 5148 <nop,nop,timestamp 680456552 892346>
13:47:42.171362 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1314809:1316257(1448) ack 1 win 1460 <nop,nop,timestamp 892370 680456552>
13:47:42.171367 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1316257:1317705(1448) ack 1 win 1460 <nop,nop,timestamp 892370 680456552>
13:47:42.171370 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 657417 win 5159 <nop,nop,timestamp 680456552 892346>
13:47:42.171376 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1317705:1319153(1448) ack 1 win 1460 <nop,nop,timestamp 892370 680456552>
13:47:42.173454 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1710113:1711561(1448) ack 1 win 1460 <nop,nop,timestamp 892371 680456553>
13:47:42.173457 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1711561:1713009(1448) ack 1 win 1460 <nop,nop,timestamp 892371 680456553>
13:47:42.173582 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 880409 win 6901 <nop,nop,timestamp 680456554 892347>
13:47:42.173591 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1721697:1723145(1448) ack 1 win 1460 <nop,nop,timestamp 892371 680456553>
13:47:42.173593 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1723145:1724593(1448) ack 1 win 1460 <nop,nop,timestamp 892371 680456553>
13:47:42.173595 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1724593:1726041(1448) ack 1 win 1460 <nop,nop,timestamp 892371 680456553>
13:47:42.173597 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1726041:1727489(1448) ack 1 win 1460 <nop,nop,timestamp 892371 680456553>
13:47:42.173599 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1727489:1728937(1448) ack 1 win 1460 <nop,nop,timestamp 892371 680456553>
13:47:42.173600 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1728937:1730385(1448) ack 1 win 1460 <nop,nop,timestamp 892371 680456553>
13:47:42.173602 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1730385:1731833(1448) ack 1 win 1460 <nop,nop,timestamp 892371 680456553>
13:47:42.173603 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1731833:1733281(1448) ack 1 win 1460 <nop,nop,timestamp 892371 680456553>
13:47:42.173605 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1733281:1734729(1448) ack 1 win 1460 <nop,nop,timestamp 892371 680456553>
13:47:42.173607 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1734729:1736177(1448) ack 1 win 1460 <nop,nop,timestamp 892371 680456553>
13:47:42.173609 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1736177:1737625(1448) ack 1 win 1460 <nop,nop,timestamp 892371 680456553>
13:47:42.173615 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 881857 win 6912 <nop,nop,timestamp 680456554 892347>
13:47:42.175340 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1905593:1907041(1448) ack 1 win 1460 <nop,nop,timestamp 892372 680456554>
13:47:42.175453 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1944689:1946137(1448) ack 1 win 1460 <nop,nop,timestamp 892372 680456554>
13:47:42.175455 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1946137:1947585(1448) ack 1 win 1460 <nop,nop,timestamp 892372 680456554>
13:47:42.175576 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1088921 win 8530 <nop,nop,timestamp 680456556 892347>
13:47:42.175587 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1957721:1959169(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456554>
13:47:42.175590 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1959169:1960617(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456554>
13:47:42.175592 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1960617:1962065(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456554>
13:47:42.175848 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1972201:1973649(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.175852 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1973649:1975097(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.175855 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1975097:1976545(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.175857 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1976545:1977993(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.175859 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1977993:1979441(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.175863 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1979441:1980889(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.175866 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1980889:1982337(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.175869 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1982337:1983785(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.175872 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1983785:1985233(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.175875 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 1985233:1986681(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.175957 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2004057:2005505(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.175960 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2005505:2006953(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.175963 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2006953:2008401(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176084 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2017089:2018537(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176087 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2018537:2019985(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176089 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2019985:2021433(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176092 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2021433:2022881(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176095 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2022881:2024329(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176097 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2024329:2025777(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176101 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2025777:2027225(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176104 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2027225:2028673(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176106 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2028673:2030121(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176109 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2030121:2031569(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176206 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2031569:2033017(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176209 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2033017:2034465(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176211 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2034465:2035913(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176214 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2035913:2037361(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176216 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2037361:2038809(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176219 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2038809:2040257(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176338 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2046049:2047497(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176342 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2047497:2048945(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176345 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2048945:2050393(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176348 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2050393:2051841(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176350 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2051841:2053289(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176353 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2053289:2054737(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176355 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2054737:2056185(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176459 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2061977:2063425(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176463 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2063425:2064873(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176466 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2064873:2066321(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176469 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2066321:2067769(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176471 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2067769:2069217(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176474 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2069217:2070665(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176477 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2070665:2072113(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176480 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2072113:2073561(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176584 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2076457:2077905(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176588 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2077905:2079353(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176590 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2079353:2080801(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176592 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2080801:2082249(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176596 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2082249:2083697(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176599 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2083697:2085145(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176602 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2085145:2086593(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176604 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2086593:2088041(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176607 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2088041:2089489(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.176609 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2089489:2090937(1448) ack 1 win 1460 <nop,nop,timestamp 892373 680456555>
13:47:42.192560 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1107745 win 8677 <nop,nop,timestamp 680456573 892368>
13:47:42.192580 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2218361:2219809(1448) ack 1 win 1460 <nop,nop,timestamp 892391 680456573>
13:47:42.192586 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2219809:2221257(1448) ack 1 win 1460 <nop,nop,timestamp 892391 680456573>
13:47:42.192589 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1109193 win 8688 <nop,nop,timestamp 680456573 892368>
13:47:42.192596 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2221257:2222705(1448) ack 1 win 1460 <nop,nop,timestamp 892391 680456573>
13:47:42.192601 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2222705:2224153(1448) ack 1 win 1460 <nop,nop,timestamp 892391 680456573>
13:47:42.192604 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1110641 win 8700 <nop,nop,timestamp 680456573 892368>
13:47:42.192610 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2224153:2225601(1448) ack 1 win 1460 <nop,nop,timestamp 892391 680456573>
13:47:42.192614 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2225601:2227049(1448) ack 1 win 1460 <nop,nop,timestamp 892391 680456573>
13:47:42.192618 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1112089 win 8711 <nop,nop,timestamp 680456573 892368>
13:47:42.192624 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2227049:2228497(1448) ack 1 win 1460 <nop,nop,timestamp 892391 680456573>
13:47:42.192629 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2228497:2229945(1448) ack 1 win 1460 <nop,nop,timestamp 892391 680456573>
13:47:42.192631 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1113537 win 8722 <nop,nop,timestamp 680456573 892368>
13:47:42.192638 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2229945:2231393(1448) ack 1 win 1460 <nop,nop,timestamp 892391 680456573>
13:47:42.192642 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2231393:2232841(1448) ack 1 win 1460 <nop,nop,timestamp 892391 680456573>
<SNIP>
# Then the following pattern will repeat itself (iperf is still happy):
13:47:42.757140 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40239945 win 18440 <nop,nop,timestamp 680457138 892932>
13:47:42.757147 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40242841 win 18440 <nop,nop,timestamp 680457138 892932>
13:47:42.757151 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40245737 win 18440 <nop,nop,timestamp 680457138 892932>
13:47:42.757156 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40248633 win 18440 <nop,nop,timestamp 680457138 892932>
13:47:42.757218 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40251529 win 18440 <nop,nop,timestamp 680457138 892932>
13:47:42.757225 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40254425 win 18440 <nop,nop,timestamp 680457138 892932>
13:47:42.757230 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40257321 win 18440 <nop,nop,timestamp 680457138 892932>
13:47:42.757240 IP bad-len 0
13:47:42.757245 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40260217 win 18440 <nop,nop,timestamp 680457138 892932>
13:47:42.757468 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40270353 win 18440 <nop,nop,timestamp 680457138 892932>
13:47:42.757476 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40273249 win 18440 <nop,nop,timestamp 680457138 892932>
13:47:42.757480 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40276145 win 18440 <nop,nop,timestamp 680457138 892932>
13:47:42.757489 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40279041 win 18440 <nop,nop,timestamp 680457138 892932>
13:47:42.757493 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40287729 win 18440 <nop,nop,timestamp 680457138 892932>
13:47:42.757595 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40290625 win 18440 <nop,nop,timestamp 680457138 892932>
13:47:42.757845 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40296417 win 18440 <nop,nop,timestamp 680457138 892933>
13:47:42.757966 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40303657 win 18440 <nop,nop,timestamp 680457138 892933>
13:47:42.758091 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40312345 win 18440 <nop,nop,timestamp 680457139 892933>
13:47:42.758097 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40315241 win 18440 <nop,nop,timestamp 680457139 892933>
13:47:42.758218 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40325377 win 18440 <nop,nop,timestamp 680457139 892933>
13:47:42.758225 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40328273 win 18440 <nop,nop,timestamp 680457139 892933>
13:47:42.758259 IP bad-len 0
13:47:42.758472 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40335513 win 18440 <nop,nop,timestamp 680457139 892933>
13:47:42.758847 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40344201 win 18440 <nop,nop,timestamp 680457139 892933>
13:47:42.758854 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40347097 win 18440 <nop,nop,timestamp 680457139 892933>
13:47:42.758859 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 40349993 win 18440 <nop,nop,timestamp 680457139 892933>
<SNIP>
# Suddenly:
13:48:04.583137 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1993873761 win 19243 <nop,nop,timestamp 680478963 914759>
13:48:04.583247 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1993883897 win 19243 <nop,nop,timestamp 680478963 914759>
13:48:04.583253 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1993886793 win 19243 <nop,nop,timestamp 680478963 914759>
13:48:04.583257 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1993889689 win 19243 <nop,nop,timestamp 680478963 914759>
13:48:04.583369 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1993898377 win 19243 <nop,nop,timestamp 680478963 914759>
13:48:04.583494 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1993907065 win 19243 <nop,nop,timestamp 680478963 914759>
13:48:04.583501 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1993909961 win 19243 <nop,nop,timestamp 680478963 914759>
13:48:04.583505 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1993912857 win 19243 <nop,nop,timestamp 680478963 914759>
13:48:04.583627 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1993922993 win 19243 <nop,nop,timestamp 680478963 914759>
13:48:04.583636 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1993925889 win 19243 <nop,nop,timestamp 680478963 914759>
13:48:04.583640 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1993928785 win 19243 <nop,nop,timestamp 680478963 914759>
13:48:04.583744 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1993937473 win 19243 <nop,nop,timestamp 680478963 914759>
13:48:04.624987 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1999274801:1999339961(65160) ack 1 win 1460 <nop,nop,timestamp 914823 680479000>
13:48:04.625480 IP bad-len 0
13:48:04.625985 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1999339961:1999405121(65160) ack 1 win 1460 <nop,nop,timestamp 914823 680479000>
13:48:04.626600 IP bad-len 0
13:48:04.627101 IP bad-len 0
13:48:04.627722 IP bad-len 0
13:48:04.628222 IP bad-len 0
13:48:04.628227 IP bad-len 0
13:48:04.628231 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 1999618073:1999683233(65160) ack 1 win 1460 <nop,nop,timestamp 914824 680479001>
13:48:04.628845 IP bad-len 0
13:48:04.629345 IP bad-len 0
13:48:04.629845 IP bad-len 0
13:48:04.630470 IP bad-len 0
13:48:04.630978 IP bad-len 0
13:48:04.630987 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2000009033:2000074193(65160) ack 1 win 1460 <nop,nop,timestamp 914825 680479003>
13:48:04.631595 IP bad-len 0
13:48:04.632218 IP bad-len 0
13:48:04.632843 IP bad-len 0
13:48:04.633343 IP bad-len 0
13:48:04.642458 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1997913681 win 19243 <nop,nop,timestamp 680479022 914800>
13:48:04.642583 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1997919473 win 19243 <nop,nop,timestamp 680479022 914820>
13:48:04.642591 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1997923817 win 19243 <nop,nop,timestamp 680479022 914820>
13:48:04.642836 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1997932505 win 19243 <nop,nop,timestamp 680479022 914820>
13:48:04.642853 IP bad-len 0
13:48:04.642961 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1997941193 win 19243 <nop,nop,timestamp 680479023 914820>
13:48:04.642970 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1997944089 win 19243 <nop,nop,timestamp 680479023 914820>
13:48:04.643084 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1997952777 win 19243 <nop,nop,timestamp 680479023 914820>
13:48:04.643092 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 1997955673 win 19243 <nop,nop,timestamp 680479023 914820>
<SNIP the ack flow continues>
# Then SACK is actionned:
13:48:07.451384 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2259565929 win 19243 <nop,nop,timestamp 680481831 917628>
13:48:07.451507 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2259574617 win 19243 <nop,nop,timestamp 680481831 917628>
13:48:07.451513 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2259577513 win 19243 <nop,nop,timestamp 680481831 917628>
13:48:07.451632 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2259586201 win 19243 <nop,nop,timestamp 680481831 917628>
13:48:07.451644 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2259589097 win 19243 <nop,nop,timestamp 680481831 917628>
13:48:07.451649 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2259591993 win 19243 <nop,nop,timestamp 680481831 917628>
13:48:07.472870 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2260166849 win 19243 <nop,nop,timestamp 680481852 917634,nop,nop,sack sack 1 {2260168297:2261533761} >
13:48:07.472880 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2260166849 win 19243 <nop,nop,timestamp 680481852 917634,nop,nop,sack sack 1 {2260168297:2261535209} >
13:48:07.472885 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2260166849 win 19243 <nop,nop,timestamp 680481852 917634,nop,nop,sack sack 1 {2260168297:2261536657} >
13:48:07.472891 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2260166849 win 19243 <nop,nop,timestamp 680481852 917634,nop,nop,sack sack 1 {2260168297:2261538105} >
13:48:07.472898 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2260166849 win 19243 <nop,nop,timestamp 680481852 917634,nop,nop,sack sack 1 {2260168297:2261539553} >
13:48:07.472903 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2260166849 win 19243 <nop,nop,timestamp 680481852 917634,nop,nop,sack sack 1 {2260168297:2261541001} >
<SNIP lots of ack with SACK>
13:48:07.474083 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2260166849 win 19243 <nop,nop,timestamp 680481854 917634,nop,nop,sack sack 1 {2260168297:2261685801} >
13:48:07.474089 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2260166849 win 19243 <nop,nop,timestamp 680481854 917634,nop,nop,sack sack 1 {2260168297:2261687249} >
13:48:07.474094 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2260166849 win 19243 <nop,nop,timestamp 680481854 917634,nop,nop,sack sack 1 {2260168297:2261688697} >
13:48:07.474099 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2260166849 win 19243 <nop,nop,timestamp 680481854 917634,nop,nop,sack sack 1 {2260168297:2261690145} >
13:48:07.474245 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2260166849 win 19243 <nop,nop,timestamp 680481854 917634,nop,nop,sack sack 1 {2260168297:2261693041} >
13:48:07.474257 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2261693041:2261694489(1448) ack 1 win 1460 <nop,nop,timestamp 917675 680481854>
13:48:07.474261 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2260166849 win 19243 <nop,nop,timestamp 680481854 917634,nop,nop,sack sack 1 {2260168297:2261694489} >
13:48:07.474271 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2261694489:2261695937(1448) ack 1 win 1460 <nop,nop,timestamp 917675 680481854>
13:48:07.474275 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2260166849 win 19243 <nop,nop,timestamp 680481854 917634,nop,nop,sack sack 1 {2260168297:2261695937} >
13:48:07.474287 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2261695937:2261697385(1448) ack 1 win 1460 <nop,nop,timestamp 917675 680481854>
13:48:07.474401 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2261710417:2261711865(1448) ack 1 win 1460 <nop,nop,timestamp 917676 680481854>
13:48:07.474413 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2261711865:2261713313(1448) ack 1 win 1460 <nop,nop,timestamp 917676 680481854>
13:48:07.474417 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2260166849 win 19243 <nop,nop,timestamp 680481854 917634,nop,nop,sack sack 1 {2260168297:2261713313} >
13:48:07.474426 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2261713313:2261714761(1448) ack 1 win 1460 <nop,nop,timestamp 917676 680481854>
13:48:07.474431 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2260166849 win 19243 <nop,nop,timestamp 680481854 917634,nop,nop,sack sack 1 {2260168297:2261714761} >
13:48:07.474440 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2261714761:2261716209(1448) ack 1 win 1460 <nop,nop,timestamp 917676 680481854>
13:48:07.474444 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2260166849 win 19243 <nop,nop,timestamp 680481854 917634,nop,nop,sack sack 1 {2260168297:2261716209} >
13:48:07.474455 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2261716209:2261717657(1448) ack 1 win 1460 <nop,nop,timestamp 917676 680481854>
13:48:07.474459 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2260166849 win 19243 <nop,nop,timestamp 680481854 917634,nop,nop,sack sack 1 {2260168297:2261717657} >
13:48:07.474468 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2261717657:2261719105(1448) ack 1 win 1460 <nop,nop,timestamp 917676 680481854>
13:48:07.474471 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2260166849 win 19243 <nop,nop,timestamp 680481854 917634,nop,nop,sack sack 1 {2260168297:2261719105} >
13:48:07.474503 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2261722001:2261723449(1448) ack 1 win 1460 <nop,nop,timestamp 917676 680481854>
13:48:07.474516 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2261723449:2261724897(1448) ack 1 win 1460 <nop,nop,timestamp 917676 680481854>
13:48:07.474520 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2260166849 win 19243 <nop,nop,timestamp 680481854 917634,nop,nop,sack sack 1 {2260168297:2261724897} >
13:48:07.474534 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2261724897:2261726345(1448) ack 1 win 1460 <nop,nop,timestamp 917676 680481854>
13:48:07.474538 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2260166849 win 19243 <nop,nop,timestamp 680481854 917634,nop,nop,sack sack 1 {2260168297:2261726345} >
13:48:07.474547 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2261726345:2261727793(1448) ack 1 win 1460 <nop,nop,timestamp 917676 680481854>
<SNIP similar flow>
Suddenly, the throughput is going down and iperf reports it (look at the time):
13:48:07.518591 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262538673 win 19243 <nop,nop,timestamp 680481898 917660,nop,nop,sack sack 1 {2262529985:2262531433} >
13:48:07.518597 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262538673 win 19243 <nop,nop,timestamp 680481898 917660,nop,nop,sack sack 1 {2262531433:2262532881} >
13:48:07.518715 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262538673 win 19243 <nop,nop,timestamp 680481898 917660,nop,nop,sack sack 1 {2262532881:2262534329} >
13:48:07.519090 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262538673 win 19243 <nop,nop,timestamp 680481899 917660,nop,nop,sack sack 1 {2262534329:2262535777} >
13:48:07.519096 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262538673 win 19243 <nop,nop,timestamp 680481899 917660,nop,nop,sack sack 1 {2262535777:2262537225} >
13:48:07.519099 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262538673 win 19243 <nop,nop,timestamp 680481899 917697,nop,nop,sack sack 1 {2262537225:2262538673} >
13:48:07.745472 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262538673:2262540121(1448) ack 1 win 1460 <nop,nop,timestamp 917947 680481899>
13:48:07.768315 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262540121 win 19243 <nop,nop,timestamp 680482148 917947>
13:48:07.995416 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262540121:2262541569(1448) ack 1 win 1460 <nop,nop,timestamp 918197 680482148>
13:48:08.018288 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262541569 win 19243 <nop,nop,timestamp 680482398 918197>
13:48:08.244357 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262541569:2262543017(1448) ack 1 win 1460 <nop,nop,timestamp 918446 680482398>
13:48:08.267138 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262543017 win 19243 <nop,nop,timestamp 680482647 918446>
13:48:08.492303 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262543017:2262544465(1448) ack 1 win 1460 <nop,nop,timestamp 918694 680482647>
13:48:08.515111 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262544465 win 19243 <nop,nop,timestamp 680482895 918694>
13:48:08.739245 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262544465:2262545913(1448) ack 1 win 1460 <nop,nop,timestamp 918941 680482895>
13:48:08.762087 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262545913 win 19243 <nop,nop,timestamp 680483142 918941>
13:48:08.986190 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262545913:2262547361(1448) ack 1 win 1460 <nop,nop,timestamp 919188 680483142>
13:48:09.009062 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262547361 win 19243 <nop,nop,timestamp 680483388 919188>
13:48:09.232135 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262547361:2262548809(1448) ack 1 win 1460 <nop,nop,timestamp 919434 680483388>
13:48:09.255164 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262548809 win 19243 <nop,nop,timestamp 680483635 919434>
13:48:09.479080 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262548809:2262550257(1448) ack 1 win 1460 <nop,nop,timestamp 919681 680483635>
13:48:09.503387 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262550257 win 19243 <nop,nop,timestamp 680483883 919681>
13:48:09.727025 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262550257:2262551705(1448) ack 1 win 1460 <nop,nop,timestamp 919929 680483883>
13:48:09.749989 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262551705 win 19243 <nop,nop,timestamp 680484129 919929>
13:48:09.973973 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262551705:2262553153(1448) ack 1 win 1460 <nop,nop,timestamp 920175 680484129>
13:48:09.996966 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262553153 win 19243 <nop,nop,timestamp 680484376 920175>
13:48:10.220915 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262553153:2262554601(1448) ack 1 win 1460 <nop,nop,timestamp 920422 680484376>
13:48:10.243690 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262554601 win 19243 <nop,nop,timestamp 680484623 920422>
13:48:10.467860 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262554601:2262556049(1448) ack 1 win 1460 <nop,nop,timestamp 920669 680484623>
13:48:10.490665 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262556049 win 19243 <nop,nop,timestamp 680484870 920669>
13:48:10.714804 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262556049:2262557497(1448) ack 1 win 1460 <nop,nop,timestamp 920916 680484870>
13:48:10.737640 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262557497 win 19243 <nop,nop,timestamp 680485117 920916>
13:48:10.960749 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262557497:2262558945(1448) ack 1 win 1460 <nop,nop,timestamp 921162 680485117>
13:48:10.983617 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262558945 win 19243 <nop,nop,timestamp 680485363 921162>
13:48:11.206695 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262558945:2262560393(1448) ack 1 win 1460 <nop,nop,timestamp 921408 680485363>
13:48:11.229968 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262560393 win 19243 <nop,nop,timestamp 680485609 921408>
13:48:11.454639 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262560393:2262561841(1448) ack 1 win 1460 <nop,nop,timestamp 921656 680485609>
13:48:11.480190 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262561841 win 19243 <nop,nop,timestamp 680485860 921656>
13:48:11.704583 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262561841:2262563289(1448) ack 1 win 1460 <nop,nop,timestamp 921906 680485860>
13:48:11.727541 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262563289 win 19243 <nop,nop,timestamp 680486107 921906>
13:48:11.951531 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262563289:2262564737(1448) ack 1 win 1460 <nop,nop,timestamp 922153 680486107>
13:48:11.974516 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262564737 win 19243 <nop,nop,timestamp 680486354 922153>
13:48:12.198473 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262564737:2262566185(1448) ack 1 win 1460 <nop,nop,timestamp 922400 680486354>
13:48:12.221367 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262566185 win 19243 <nop,nop,timestamp 680486601 922400>
13:48:12.444424 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262566185:2262567633(1448) ack 1 win 1460 <nop,nop,timestamp 922646 680486601>
13:48:12.467343 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262567633 win 19243 <nop,nop,timestamp 680486847 922646>
13:48:12.690364 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262567633:2262569081(1448) ack 1 win 1460 <nop,nop,timestamp 922892 680486847>
13:48:12.713318 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262569081 win 19243 <nop,nop,timestamp 680487093 922892>
13:48:12.936310 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262569081:2262570529(1448) ack 1 win 1460 <nop,nop,timestamp 923138 680487093>
13:48:12.959171 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262570529 win 19243 <nop,nop,timestamp 680487339 923138>
13:48:13.182254 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262570529:2262571977(1448) ack 1 win 1460 <nop,nop,timestamp 923384 680487339>
13:48:13.205145 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262571977 win 19243 <nop,nop,timestamp 680487584 923384>
13:48:13.428197 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262571977:2262573425(1448) ack 1 win 1460 <nop,nop,timestamp 923630 680487584>
13:48:13.451371 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262573425 win 19243 <nop,nop,timestamp 680487831 923630>
13:48:13.674143 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262573425:2262574873(1448) ack 1 win 1460 <nop,nop,timestamp 923876 680487831>
13:48:13.696972 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262574873 win 19243 <nop,nop,timestamp 680488076 923876>
13:48:13.920093 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262574873:2262576321(1448) ack 1 win 1460 <nop,nop,timestamp 924122 680488076>
13:48:13.942949 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262576321 win 19243 <nop,nop,timestamp 680488322 924122>
13:48:14.166033 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262576321:2262577769(1448) ack 1 win 1460 <nop,nop,timestamp 924368 680488322>
13:48:14.188926 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262577769 win 19243 <nop,nop,timestamp 680488568 924368>
13:48:14.411981 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262577769:2262579217(1448) ack 1 win 1460 <nop,nop,timestamp 924614 680488568>
13:48:14.434901 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262579217 win 19243 <nop,nop,timestamp 680488814 924614>
13:48:14.657924 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262579217:2262580665(1448) ack 1 win 1460 <nop,nop,timestamp 924860 680488814>
13:48:14.680876 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262580665 win 19243 <nop,nop,timestamp 680489060 924860>
13:48:14.903868 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262580665:2262582113(1448) ack 1 win 1460 <nop,nop,timestamp 925106 680489060>
13:48:14.926727 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262582113 win 19243 <nop,nop,timestamp 680489306 925106>
13:48:15.149818 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262582113:2262583561(1448) ack 1 win 1460 <nop,nop,timestamp 925352 680489306>
13:48:15.172953 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262583561 win 19243 <nop,nop,timestamp 680489552 925352>
13:48:15.395759 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262583561:2262585009(1448) ack 1 win 1460 <nop,nop,timestamp 925598 680489552>
13:48:15.418680 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262585009 win 19243 <nop,nop,timestamp 680489798 925598>
13:48:15.641704 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262585009:2262586457(1448) ack 1 win 1460 <nop,nop,timestamp 925844 680489798>
13:48:15.664530 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262586457 win 19243 <nop,nop,timestamp 680490044 925844>
13:48:15.887653 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262586457:2262587905(1448) ack 1 win 1460 <nop,nop,timestamp 926090 680490044>
13:48:15.910632 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2262587905 win 19243 <nop,nop,timestamp 680490290 926090>
13:48:16.133595 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2262587905:2262589353(1448) ack 1 win 1460 <nop,nop,timestamp 926336 680490290>
<SNIP similar traffic>
# Then suddenly the throughput is starting again at 13:51:14!
13:51:10.827587 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263688385 win 19243 <nop,nop,timestamp 680665200 1101027>
13:51:11.050591 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2263688385:2263689833(1448) ack 1 win 1460 <nop,nop,timestamp 1101273 680665200>
13:51:11.073563 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263689833 win 19243 <nop,nop,timestamp 680665446 1101273>
13:51:11.296536 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2263689833:2263691281(1448) ack 1 win 1460 <nop,nop,timestamp 1101519 680665446>
13:51:11.319414 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263691281 win 19243 <nop,nop,timestamp 680665692 1101519>
13:51:11.542483 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2263691281:2263692729(1448) ack 1 win 1460 <nop,nop,timestamp 1101765 680665692>
13:51:11.565516 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263692729 win 19243 <nop,nop,timestamp 680665938 1101765>
13:51:11.788427 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2263692729:2263694177(1448) ack 1 win 1460 <nop,nop,timestamp 1102011 680665938>
13:51:11.811242 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263694177 win 19243 <nop,nop,timestamp 680666184 1102011>
13:51:12.034372 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2263694177:2263695625(1448) ack 1 win 1460 <nop,nop,timestamp 1102257 680666184>
13:51:12.057218 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263695625 win 19243 <nop,nop,timestamp 680666430 1102257>
13:51:12.280317 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2263695625:2263697073(1448) ack 1 win 1460 <nop,nop,timestamp 1102503 680666430>
13:51:12.303069 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263697073 win 19243 <nop,nop,timestamp 680666676 1102503>
13:51:12.526273 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2263697073:2263698521(1448) ack 1 win 1460 <nop,nop,timestamp 1102749 680666676>
13:51:12.549296 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263698521 win 19243 <nop,nop,timestamp 680666922 1102749>
13:51:12.772208 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2263698521:2263699969(1448) ack 1 win 1460 <nop,nop,timestamp 1102995 680666922>
13:51:12.795146 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263699969 win 19243 <nop,nop,timestamp 680667168 1102995>
13:51:13.018155 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2263699969:2263701417(1448) ack 1 win 1460 <nop,nop,timestamp 1103241 680667168>
13:51:13.041122 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263701417 win 19243 <nop,nop,timestamp 680667413 1103241>
13:51:13.264099 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2263701417:2263702865(1448) ack 1 win 1460 <nop,nop,timestamp 1103487 680667413>
13:51:13.286974 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263702865 win 19243 <nop,nop,timestamp 680667659 1103487>
13:51:13.510044 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2263702865:2263704313(1448) ack 1 win 1460 <nop,nop,timestamp 1103733 680667659>
13:51:13.538322 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263704313 win 19243 <nop,nop,timestamp 680667910 1103733>
13:51:13.761992 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2263704313:2263705761(1448) ack 1 win 1460 <nop,nop,timestamp 1103985 680667910>
13:51:13.784923 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263705761 win 19243 <nop,nop,timestamp 680668157 1103985>
13:51:14.008935 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2263705761:2263707209(1448) ack 1 win 1460 <nop,nop,timestamp 1104232 680668157>
13:51:14.031773 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263707209 win 19243 <nop,nop,timestamp 680668404 1104232>
13:51:14.031782 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263707209:2263708657(1448) ack 1 win 1460 <nop,nop,timestamp 1104255 680668404>
13:51:14.031787 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263708657:2263710105(1448) ack 1 win 1460 <nop,nop,timestamp 1104255 680668404>
13:51:14.031791 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: P 2263710105:2263711553(1448) ack 1 win 1460 <nop,nop,timestamp 1104255 680668404>
13:51:14.054759 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263708657 win 19243 <nop,nop,timestamp 680668427 1104255>
13:51:14.054768 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263711553:2263713001(1448) ack 1 win 1460 <nop,nop,timestamp 1104278 680668427>
13:51:14.054773 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263713001:2263714449(1448) ack 1 win 1460 <nop,nop,timestamp 1104278 680668427>
13:51:14.054776 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263710105 win 19243 <nop,nop,timestamp 680668427 1104255>
13:51:14.054783 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263714449:2263715897(1448) ack 1 win 1460 <nop,nop,timestamp 1104278 680668427>
13:51:14.054787 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263715897:2263717345(1448) ack 1 win 1460 <nop,nop,timestamp 1104278 680668427>
13:51:14.054790 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263711553 win 19243 <nop,nop,timestamp 680668427 1104255>
13:51:14.054796 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263717345:2263718793(1448) ack 1 win 1460 <nop,nop,timestamp 1104278 680668427>
13:51:14.054801 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263718793:2263720241(1448) ack 1 win 1460 <nop,nop,timestamp 1104278 680668427>
13:51:14.077746 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263713001 win 19243 <nop,nop,timestamp 680668450 1104278>
13:51:14.077755 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263720241:2263721689(1448) ack 1 win 1460 <nop,nop,timestamp 1104301 680668450>
13:51:14.077759 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263721689:2263723137(1448) ack 1 win 1460 <nop,nop,timestamp 1104301 680668450>
13:51:14.077762 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263714449 win 19243 <nop,nop,timestamp 680668450 1104278>
13:51:14.077767 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263723137:2263724585(1448) ack 1 win 1460 <nop,nop,timestamp 1104301 680668450>
13:51:14.077771 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263724585:2263726033(1448) ack 1 win 1460 <nop,nop,timestamp 1104301 680668450>
13:51:14.077774 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263715897 win 19243 <nop,nop,timestamp 680668450 1104278>
13:51:14.077781 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263726033:2263727481(1448) ack 1 win 1460 <nop,nop,timestamp 1104301 680668450>
13:51:14.077785 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263727481:2263728929(1448) ack 1 win 1460 <nop,nop,timestamp 1104301 680668450>
13:51:14.077788 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263717345 win 19243 <nop,nop,timestamp 680668450 1104278>
13:51:14.077793 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263728929:2263730377(1448) ack 1 win 1460 <nop,nop,timestamp 1104301 680668450>
13:51:14.077797 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263730377:2263731825(1448) ack 1 win 1460 <nop,nop,timestamp 1104301 680668450>
13:51:14.077801 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263718793 win 19243 <nop,nop,timestamp 680668450 1104278>
13:51:14.077807 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263731825:2263733273(1448) ack 1 win 1460 <nop,nop,timestamp 1104301 680668450>
13:51:14.077811 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263733273:2263734721(1448) ack 1 win 1460 <nop,nop,timestamp 1104301 680668450>
13:51:14.077814 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263720241 win 19243 <nop,nop,timestamp 680668450 1104278>
13:51:14.077819 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263734721:2263736169(1448) ack 1 win 1460 <nop,nop,timestamp 1104301 680668450>
13:51:14.077823 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263736169:2263737617(1448) ack 1 win 1460 <nop,nop,timestamp 1104301 680668450>
13:51:14.100731 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263721689 win 19243 <nop,nop,timestamp 680668473 1104301>
13:51:14.100741 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263737617:2263739065(1448) ack 1 win 1460 <nop,nop,timestamp 1104324 680668473>
13:51:14.100746 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263739065:2263740513(1448) ack 1 win 1460 <nop,nop,timestamp 1104324 680668473>
13:51:14.100749 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263723137 win 19243 <nop,nop,timestamp 680668473 1104301>
13:51:14.100755 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263740513:2263741961(1448) ack 1 win 1460 <nop,nop,timestamp 1104324 680668473>
13:51:14.100759 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263741961:2263743409(1448) ack 1 win 1460 <nop,nop,timestamp 1104324 680668473>
13:51:14.100762 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263724585 win 19243 <nop,nop,timestamp 680668473 1104301>
13:51:14.100767 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263743409:2263744857(1448) ack 1 win 1460 <nop,nop,timestamp 1104324 680668473>
13:51:14.100772 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263744857:2263746305(1448) ack 1 win 1460 <nop,nop,timestamp 1104324 680668473>
13:51:14.100859 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263726033 win 19243 <nop,nop,timestamp 680668473 1104301>
13:51:14.100866 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263746305:2263747753(1448) ack 1 win 1460 <nop,nop,timestamp 1104324 680668473>
13:51:14.100871 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263747753:2263749201(1448) ack 1 win 1460 <nop,nop,timestamp 1104324 680668473>
13:51:14.100874 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263727481 win 19243 <nop,nop,timestamp 680668473 1104301>
13:51:14.100879 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263749201:2263750649(1448) ack 1 win 1460 <nop,nop,timestamp 1104324 680668473>
13:51:14.100884 IP lxshare220d.cern.ch.32772 > Rembrandt3.grid.sara.nl.5001: . 2263750649:2263752097(1448) ack 1 win 1460 <nop,nop,timestamp 1104324 680668473>
13:51:14.100886 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32772: . ack 2263728929 win 19243 <nop,nop,timestamp 680668473 1104301>
<SNIP throughput now stable is acceptable>

12:40

TCPDUMP of the problem:

11:59:25.773605 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2276281:2277729(1448) ack 1 win 1460 <nop,nop,timestamp 8878908 673960404>
11:59:25.773609 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2277729:2279177(1448) ack 1 win 1460 <nop,nop,timestamp 8878908 673960404>
11:59:25.773612 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1138153 win 8915 <nop,nop,timestamp 673960404 8878885>
11:59:25.773618 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 2279177:2280625(1448) ack 1 win 1460 <nop,nop,timestamp 8878908 673960404>
11:59:25.773622 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2280625:2282073(1448) ack 1 win 1460 <nop,nop,timestamp 8878908 673960404>
11:59:25.774422 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1158425 win 9073 <nop,nop,timestamp 673960404 8878885>
11:59:25.774435 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2319721:2321169(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774440 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 2321169:2322617(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774443 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1159873 win 9084 <nop,nop,timestamp 673960404 8878885>
11:59:25.774450 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2322617:2324065(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774455 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2324065:2325513(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774457 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1161321 win 9096 <nop,nop,timestamp 673960404 8878885>
11:59:25.774467 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2325513:2326961(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774472 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2326961:2328409(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774475 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1162769 win 9107 <nop,nop,timestamp 673960404 8878885>
11:59:25.774481 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 2328409:2329857(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774485 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2329857:2331305(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774488 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1164217 win 9118 <nop,nop,timestamp 673960404 8878885>
11:59:25.774495 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2331305:2332753(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774498 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 2332753:2334201(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
<SNIP>
11:59:25.774542 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1170009 win 9164 <nop,nop,timestamp 673960404 8878885>
11:59:25.774548 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2342889:2344337(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774552 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2344337:2345785(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774555 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1171457 win 9175 <nop,nop,timestamp 673960404 8878885>
11:59:25.774561 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 2345785:2347233(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774565 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2347233:2348681(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774568 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1172905 win 9186 <nop,nop,timestamp 673960404 8878885>
11:59:25.774574 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2348681:2350129(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774578 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2350129:2351577(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774581 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1174353 win 9198 <nop,nop,timestamp 673960404 8878885>
11:59:25.774587 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2351577:2353025(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774591 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 2353025:2354473(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774594 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1175801 win 9209 <nop,nop,timestamp 673960404 8878885>
11:59:25.774600 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2354473:2355921(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774604 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2355921:2357369(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774607 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1177249 win 9220 <nop,nop,timestamp 673960404 8878885>
11:59:25.774612 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 2357369:2358817(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774617 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2358817:2360265(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774620 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1178697 win 9231 <nop,nop,timestamp 673960404 8878885>
11:59:25.774629 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2360265:2361713(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774633 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 2361713:2363161(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774636 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1180145 win 9243 <nop,nop,timestamp 673960404 8878885>
11:59:25.774642 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2363161:2364609(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774646 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2364609:2366057(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774649 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1181593 win 9254 <nop,nop,timestamp 673960404 8878885>
11:59:25.774655 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 2366057:2367505(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774659 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2367505:2368953(1448) ack 1 win 1460 <nop,nop,timestamp 8878909 673960404>
11:59:25.774662 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1183041 win 9265 <nop,nop,timestamp 673960404 8878885>
11:59:25.775473 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1251097 win 9797 <nop,nop,timestamp 673960405 8878885>
11:59:25.775482 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2490585:2492033(1448) ack 1 win 1460 <nop,nop,timestamp 8878910 673960405>
11:59:25.775485 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 2492033:2493481(1448) ack 1 win 1460 <nop,nop,timestamp 8878910 673960405>
11:59:25.775487 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2493481:2494929(1448) ack 1 win 1460 <nop,nop,timestamp 8878910 673960405>
11:59:25.775489 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2494929:2496377(1448) ack 1 win 1460 <nop,nop,timestamp 8878910 673960405>
11:59:25.775490 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 2496377:2497825(1448) ack 1 win 1460 <nop,nop,timestamp 8878910 673960405>
11:59:25.775492 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2497825:2499273(1448) ack 1 win 1460 <nop,nop,timestamp 8878910 673960405>
11:59:25.775494 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2499273:2500721(1448) ack 1 win 1460 <nop,nop,timestamp 8878910 673960405>
11:59:25.775495 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 2500721:2502169(1448) ack 1 win 1460 <nop,nop,timestamp 8878910 673960405>
11:59:25.775497 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2502169:2503617(1448) ack 1 win 1460 <nop,nop,timestamp 8878910 673960405>
11:59:25.775499 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2503617:2505065(1448) ack 1 win 1460 <nop,nop,timestamp 8878910 673960405>
11:59:25.775500 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 2505065:2506513(1448) ack 1 win 1460 <nop,nop,timestamp 8878910 673960405>
11:59:25.775505 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2506513:2507961(1448) ack 1 win 1460 <nop,nop,timestamp 8878910 673960405>
11:59:25.775508 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1252545 win 9808 <nop,nop,timestamp 673960405 8878885>
11:59:25.775514 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2507961:2509409(1448) ack 1 win 1460 <nop,nop,timestamp 8878910 673960405>
11:59:25.777842 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2809145:2810593(1448) ack 1 win 1460 <nop,nop,timestamp 8878911 673960406>
11:59:25.777846 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2810593:2812041(1448) ack 1 win 1460 <nop,nop,timestamp 8878911 673960406>
11:59:25.777848 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 2812041:2813489(1448) ack 1 win 1460 <nop,nop,timestamp 8878911 673960406>
11:59:25.778051 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2916297:2917745(1448) ack 1 win 1460 <nop,nop,timestamp 8878912 673960407>
11:59:25.778053 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2917745:2919193(1448) ack 1 win 1460 <nop,nop,timestamp 8878912 673960407>
11:59:25.778055 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 2919193:2920641(1448) ack 1 win 1460 <nop,nop,timestamp 8878912 673960407>
11:59:25.778057 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2920641:2922089(1448) ack 1 win 1460 <nop,nop,timestamp 8878912 673960407>
11:59:25.778059 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2922089:2923537(1448) ack 1 win 1460 <nop,nop,timestamp 8878912 673960407>
11:59:25.778193 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1513185 win 11845 <nop,nop,timestamp 673960408 8878887>
11:59:25.778210 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 2926433:2927881(1448) ack 1 win 1460 <nop,nop,timestamp 8878912 673960407>
11:59:25.778213 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2927881:2929329(1448) ack 1 win 1460 <nop,nop,timestamp 8878912 673960407>
11:59:25.778215 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2929329:2930777(1448) ack 1 win 1460 <nop,nop,timestamp 8878912 673960407>
11:59:25.778217 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 2930777:2932225(1448) ack 1 win 1460 <nop,nop,timestamp 8878912 673960407>
11:59:25.778219 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2932225:2933673(1448) ack 1 win 1460 <nop,nop,timestamp 8878912 673960407>
11:59:25.778221 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2933673:2935121(1448) ack 1 win 1460 <nop,nop,timestamp 8878912 673960407>
11:59:25.778222 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 2935121:2936569(1448) ack 1 win 1460 <nop,nop,timestamp 8878912 673960407>
11:59:25.778224 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2936569:2938017(1448) ack 1 win 1460 <nop,nop,timestamp 8878912 673960407>
11:59:25.778226 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 2938017:2939465(1448) ack 1 win 1460 <nop,nop,timestamp 8878912 673960407>
11:59:25.778233 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 2939465:2940913(1448) ack 1 win 1460 <nop,nop,timestamp 8878912 673960407>
11:59:25.778240 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1514633 win 11856 <nop,nop,timestamp 673960408 8878887>
11:59:25.778252 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1516081 win 11867 <nop,nop,timestamp 673960408 8878887>
11:59:25.779427 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1723145 win 13485 <nop,nop,timestamp 673960410 8878889>
11:59:25.779434 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1724593 win 13496 <nop,nop,timestamp 673960410 8878889>
11:59:25.779439 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1726041 win 13508 <nop,nop,timestamp 673960410 8878889>
11:59:25.779443 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1727489 win 13519 <nop,nop,timestamp 673960410 8878889>
11:59:25.779447 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1728937 win 13530 <nop,nop,timestamp 673960410 8878889>
11:59:25.779452 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1730385 win 13542 <nop,nop,timestamp 673960410 8878889>
11:59:25.779690 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1737625 win 13598 <nop,nop,timestamp 673960410 8878889>
11:59:25.779697 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1739073 win 13609 <nop,nop,timestamp 673960410 8878889>
11:59:25.779702 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1740521 win 13621 <nop,nop,timestamp 673960410 8878889>
11:59:25.779706 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1741969 win 13632 <nop,nop,timestamp 673960410 8878889>
11:59:25.779800 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1743417 win 13643 <nop,nop,timestamp 673960410 8878889>
11:59:25.779805 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1744865 win 13655 <nop,nop,timestamp 673960410 8878889>
11:59:25.779809 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1746313 win 13666 <nop,nop,timestamp 673960410 8878889>
11:59:25.779814 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1747761 win 13677 <nop,nop,timestamp 673960410 8878889>
11:59:25.779818 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1749209 win 13689 <nop,nop,timestamp 673960410 8878889>
11:59:25.779822 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1750657 win 13700 <nop,nop,timestamp 673960410 8878889>
11:59:25.779826 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1752105 win 13711 <nop,nop,timestamp 673960410 8878889>
11:59:25.779833 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1753553 win 13723 <nop,nop,timestamp 673960410 8878889>
11:59:25.779837 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1755001 win 13734 <nop,nop,timestamp 673960410 8878889>
11:59:25.779842 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1756449 win 13745 <nop,nop,timestamp 673960410 8878889>
11:59:25.779846 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1757897 win 13756 <nop,nop,timestamp 673960410 8878889>
11:59:25.779850 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1759345 win 13768 <nop,nop,timestamp 673960410 8878889>
11:59:25.779854 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1760793 win 13779 <nop,nop,timestamp 673960410 8878889>
11:59:25.779858 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1762241 win 13790 <nop,nop,timestamp 673960410 8878889>
11:59:25.779929 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1763689 win 13802 <nop,nop,timestamp 673960410 8878889>
11:59:25.779935 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1765137 win 13813 <nop,nop,timestamp 673960410 8878889>
11:59:25.779939 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1766585 win 13824 <nop,nop,timestamp 673960410 8878889>
11:59:25.779943 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1768033 win 13836 <nop,nop,timestamp 673960410 8878889>
11:59:25.780174 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1792649 win 14028 <nop,nop,timestamp 673960410 8878889>
11:59:25.780181 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1794097 win 14039 <nop,nop,timestamp 673960410 8878889>
11:59:25.780194 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1795545 win 14051 <nop,nop,timestamp 673960410 8878889>
11:59:25.780198 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1796993 win 14062 <nop,nop,timestamp 673960410 8878889>
11:59:25.780202 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1798441 win 14073 <nop,nop,timestamp 673960410 8878889>
11:59:25.780207 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1799889 win 14085 <nop,nop,timestamp 673960410 8878889>
11:59:25.780211 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1801337 win 14096 <nop,nop,timestamp 673960410 8878889>
11:59:25.780215 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1802785 win 14107 <nop,nop,timestamp 673960410 8878889>
11:59:25.780434 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1804233 win 14118 <nop,nop,timestamp 673960411 8878889>
11:59:25.780443 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1805681 win 14130 <nop,nop,timestamp 673960411 8878889>
11:59:25.780448 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1807129 win 14141 <nop,nop,timestamp 673960411 8878889>
11:59:25.780452 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1808577 win 14152 <nop,nop,timestamp 673960411 8878889>
11:59:25.780456 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1810025 win 14164 <nop,nop,timestamp 673960411 8878889>
11:59:25.780461 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1811473 win 14175 <nop,nop,timestamp 673960411 8878889>
11:59:25.780465 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1812921 win 14186 <nop,nop,timestamp 673960411 8878889>
11:59:25.780469 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1814369 win 14198 <nop,nop,timestamp 673960411 8878889>
11:59:25.780473 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1815817 win 14209 <nop,nop,timestamp 673960411 8878889>
11:59:25.780478 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1817265 win 14220 <nop,nop,timestamp 673960411 8878889>
11:59:25.780482 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1818713 win 14232 <nop,nop,timestamp 673960411 8878889>
11:59:25.780487 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1820161 win 14243 <nop,nop,timestamp 673960411 8878890>
11:59:25.780492 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1821609 win 14254 <nop,nop,timestamp 673960411 8878890>
11:59:25.780499 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1823057 win 14266 <nop,nop,timestamp 673960411 8878890>
11:59:25.780503 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1824505 win 14277 <nop,nop,timestamp 673960411 8878890>
11:59:25.780507 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1825953 win 14288 <nop,nop,timestamp 673960411 8878890>
11:59:25.780675 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1827401 win 14299 <nop,nop,timestamp 673960411 8878890>
11:59:25.780681 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1828849 win 14311 <nop,nop,timestamp 673960411 8878890>
11:59:25.780686 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1830297 win 14322 <nop,nop,timestamp 673960411 8878890>
11:59:25.780690 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1831745 win 14333 <nop,nop,timestamp 673960411 8878890>
11:59:25.780694 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1833193 win 14345 <nop,nop,timestamp 673960411 8878890>
11:59:25.780698 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1834641 win 14356 <nop,nop,timestamp 673960411 8878890>
11:59:25.780702 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1836089 win 14367 <nop,nop,timestamp 673960411 8878890>
11:59:25.780707 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1837537 win 14379 <nop,nop,timestamp 673960411 8878890>
11:59:25.780711 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1838985 win 14390 <nop,nop,timestamp 673960411 8878890>
11:59:25.780719 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1840433 win 14401 <nop,nop,timestamp 673960411 8878890>
11:59:25.780723 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1841881 win 14413 <nop,nop,timestamp 673960411 8878890>
11:59:25.780727 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1843329 win 14424 <nop,nop,timestamp 673960411 8878890>
11:59:25.780731 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1844777 win 14435 <nop,nop,timestamp 673960411 8878890>
11:59:25.780736 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1846225 win 14447 <nop,nop,timestamp 673960411 8878890>
11:59:25.780740 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1847673 win 14458 <nop,nop,timestamp 673960411 8878890>
11:59:25.780920 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1856361 win 14526 <nop,nop,timestamp 673960411 8878890>
11:59:25.780929 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1857809 win 14537 <nop,nop,timestamp 673960411 8878890>
<SNIP>
11:59:25.800683 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 2629593 win 18440 <nop,nop,timestamp 673960431 8878910>
11:59:25.800688 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 2632489 win 18440 <nop,nop,timestamp 673960431 8878910>
11:59:25.800694 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 2635385 win 18440 <nop,nop,timestamp 673960431 8878910>
11:59:25.800905 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 2645521 win 18440 <nop,nop,timestamp 673960431 8878910>
11:59:25.800919 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 2648417 win 18440 <nop,nop,timestamp 673960431 8878910>
11:59:25.800924 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 2651313 win 18440 <nop,nop,timestamp 673960431 8878910>
11:59:25.801281 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 2660001 win 18440 <nop,nop,timestamp 673960431 8878910>
11:59:25.801296 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 2662897 win 18440 <nop,nop,timestamp 673960431 8878911>
11:59:25.801301 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 2665793 win 18440 <nop,nop,timestamp 673960431 8878911>
11:59:25.801306 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 2668689 win 18440 <nop,nop,timestamp 673960431 8878911>
11:59:25.801405 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 2678825 win 18440 <nop,nop,timestamp 673960432 8878911>
11:59:25.801417 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 2681721 win 18440 <nop,nop,timestamp 673960432 8878911>
11:59:25.801422 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 2684617 win 18440 <nop,nop,timestamp 673960432 8878911>
11:59:25.801426 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 2687513 win 18440 <nop,nop,timestamp 673960432 8878911>
11:59:25.801431 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 2690409 win 18440 <nop,nop,timestamp 673960432 8878911>
11:59:25.801441 IP bad-len 0
11:59:25.801446 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 2693305 win 18440 <nop,nop,timestamp 673960432 8878911>
<SNIP (long time) Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack xxxxx win 18440>
11:59:29.022455 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 256843369 win 18440 <nop,nop,timestamp 673963652 8882130>
11:59:29.022461 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 256846265 win 18440 <nop,nop,timestamp 673963652 8882130>
11:59:29.022467 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 256849161 win 18440 <nop,nop,timestamp 673963652 8882130>
11:59:29.022473 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 256852057 win 18440 <nop,nop,timestamp 673963652 8882130>
11:59:29.022479 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 256854953 win 18440 <nop,nop,timestamp 673963652 8882130>
11:59:29.022581 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 256863641 win 18440 <nop,nop,timestamp 673963653 8882130>
11:59:29.022587 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 256866537 win 18440 <nop,nop,timestamp 673963653 8882130>
11:59:29.022610 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: . 259097905:259163065(65160) ack 1 win 1460 <nop,nop,timestamp 8882157 673963653>
11:59:29.022834 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 256876673 win 18440 <nop,nop,timestamp 673963653 8882130>
11:59:29.022843 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 256879569 win 18440 <nop,nop,timestamp 673963653 8882130>
11:59:29.022847 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 256882465 win 18440 <nop,nop,timestamp 673963653 8882130>
11:59:29.022954 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 256891153 win 18440 <nop,nop,timestamp 673963653 8882131>
11:59:29.022960 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 256894049 win 18440 <nop,nop,timestamp 673963653 8882131>
11:59:29.023084 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 256902737 win 18440 <nop,nop,timestamp 673963653 8882131>
11:59:29.023092 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 256905633 win 18440 <nop,nop,timestamp 673963653 8882131>
11:59:29.023096 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 256908529 win 18440 <nop,nop,timestamp 673963653 8882131>
11:59:29.023328 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 256917217 win 18440 <nop,nop,timestamp 673963653 8882131>
11:59:29.023335 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 256920113 win 18440 <nop,nop,timestamp 673963653 8882131>
<SNIP In the middle of the many, many ack, one lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001 is sent (around every sec)>
:1659466057} >
11:59:45.902240 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1659479089 win 23157 <nop,nop,timestamp 673980532 8898978,nop,nop,sack sack 1 {1659466057:1659467505} >
11:59:45.902243 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1659479089 win 23157 <nop,nop,timestamp 673980532 8898978,nop,nop,sack sack 1 {1659467505:1659468953} >
11:59:45.902247 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1659479089 win 23157 <nop,nop,timestamp 673980532 8898978,nop,nop,sack sack 1 {1659468953:1659470401} >
11:59:45.903605 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1659479089 win 23157 <nop,nop,timestamp 673980533 8898978,nop,nop,sack sack 1 {1659470401:1659471849} >
11:59:45.903611 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1659479089 win 23157 <nop,nop,timestamp 673980533 8898978,nop,nop,sack sack 1 {1659471849:1659473297} >
11:59:45.903614 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1659479089 win 23157 <nop,nop,timestamp 673980533 8898978,nop,nop,sack sack 1 {1659473297:1659474745} >
11:59:45.903618 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1659479089 win 23157 <nop,nop,timestamp 673980533 8898978,nop,nop,sack sack 1 {1659474745:1659476193} >
11:59:45.903621 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1659479089 win 23157 <nop,nop,timestamp 673980533 8898978,nop,nop,sack sack 1 {1659476193:1659477641} >
11:59:45.903625 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1659479089 win 23157 <nop,nop,timestamp 673980533 8899017,nop,nop,sack sack 1 {1659477641:1659479089} >
11:59:46.134717 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 1659479089:1659480537(1448) ack 1 win 1460 <nop,nop,timestamp 8899271 673980533>
11:59:46.157701 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1659480537 win 23157 <nop,nop,timestamp 673980787 8899271>
11:59:46.388636 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 1659480537:1659481985(1448) ack 1 win 1460 <nop,nop,timestamp 8899525 673980787>
11:59:46.411546 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1659481985 win 23157 <nop,nop,timestamp 673981041 8899525>
11:59:46.640611 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 1659481985:1659483433(1448) ack 1 win 1460 <nop,nop,timestamp 8899777 673981041>
11:59:46.664022 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1659483433 win 23157 <nop,nop,timestamp 673981293 8899777>
11:59:46.892526 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 1659483433:1659484881(1448) ack 1 win 1460 <nop,nop,timestamp 8900029 673981293>
11:59:46.915992 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1659484881 win 23157 <nop,nop,timestamp 673981545 8900029>
11:59:47.144468 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 1659484881:1659486329(1448) ack 1 win 1460 <nop,nop,timestamp 8900281 673981545>
11:59:47.167839 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1659486329 win 23157 <nop,nop,timestamp 673981797 8900281>
11:59:47.395411 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 1659486329:1659487777(1448) ack 1 win 1460 <nop,nop,timestamp 8900532 673981797>
11:59:47.418436 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1659487777 win 23157 <nop,nop,timestamp 673982048 8900532>
11:59:47.645356 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 1659487777:1659489225(1448) ack 1 win 1460 <nop,nop,timestamp 8900782 673982048>
11:59:47.668410 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1659489225 win 23157 <nop,nop,timestamp 673982298 8900782>
11:59:47.894300 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 1659489225:1659490673(1448) ack 1 win 1460 <nop,nop,timestamp 8901031 673982298>
11:59:47.917384 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1659490673 win 23157 <nop,nop,timestamp 673982547 8901031>
11:59:48.143245 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 1659490673:1659492121(1448) ack 1 win 1460 <nop,nop,timestamp 8901280 673982547>
11:59:48.167235 IP Rembrandt3.grid.sara.nl.5001 > lxshare220d.cern.ch.32782: . ack 1659492121 win 23157 <nop,nop,timestamp 673982796 8901280>
11:59:48.393188 IP lxshare220d.cern.ch.32782 > Rembrandt3.grid.sara.nl.5001: P 1659492121:1659493569(1448) ack 1 win 1460 <nop,nop,timestamp 8901530 673982796>
<SNIP Throughput is now normal>

11:40

The problem is replicable with only one parameter. But it is intermittent.

9:20

The problem is replicable with only one change in the TCP parameters:

net.core.wmem_max = 134217728

It is necessary to increase the wmem_max size to use TCP window size bigger than the default 256K. The rest is all default SLC4 with up-to-date e1000 driver.

[root@lxshare220d ~]# iperf -c rembrandt3.grid.sara.nl  -t 12000 -i 30 -w 2M -p5001 | tee freshfromreboot_defaultssyctl.txt
------------------------------------------------------------
Client connecting to rembrandt3.grid.sara.nl, TCP port 5001
TCP window size: 4.00 MByte (WARNING: requested 2.00 MByte)
------------------------------------------------------------
[  3] local 128.142.224.220 port 32780 connected with 145.100.3.85 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-30.0 sec   801 MBytes   224 Mbits/sec
[  3] 30.0-60.0 sec   153 MBytes  42.9 Mbits/sec
[  3] 60.0-90.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 90.0-120.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 120.0-150.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 150.0-180.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 180.0-210.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 210.0-240.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 240.0-270.0 sec  1.01 MBytes   282 Kbits/sec
[  3] 270.0-300.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 300.0-330.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 330.0-360.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 360.0-390.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 390.0-420.0 sec  84.9 MBytes  23.7 Mbits/sec
[  3] 420.0-450.0 sec  2.51 GBytes   719 Mbits/sec
[  3] 450.0-480.0 sec  2.33 GBytes   668 Mbits/sec
[  3] 480.0-510.0 sec  2.53 GBytes   724 Mbits/sec
[  3] 510.0-540.0 sec  2.61 GBytes   748 Mbits/sec
[  3] 540.0-570.0 sec  2.44 GBytes   699 Mbits/sec
[  3] 570.0-600.0 sec  2.61 GBytes   748 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3] 600.0-630.0 sec  2.54 GBytes   727 Mbits/sec
[  3] 630.0-660.0 sec  2.52 GBytes   723 Mbits/sec
[  3] 660.0-690.0 sec  2.61 GBytes   747 Mbits/sec
[  3] 690.0-720.0 sec  2.68 GBytes   768 Mbits/sec
[  3] 720.0-750.0 sec  2.61 GBytes   747 Mbits/sec
[  3] 750.0-780.0 sec  2.33 GBytes   668 Mbits/sec
[  3] 780.0-810.0 sec  2.42 GBytes   694 Mbits/sec
[  3] 810.0-840.0 sec  2.47 GBytes   707 Mbits/sec
[  3] 840.0-870.0 sec  2.40 GBytes   686 Mbits/sec
[  3] 870.0-900.0 sec  2.44 GBytes   700 Mbits/sec
[  3] 900.0-930.0 sec  2.42 GBytes   693 Mbits/sec

10th November 2005

15:00

iptables has been unloaded, no changes in the behavior. However, it is replicable to different sites:

[root@lxshare025d ~]# iperf  -c rembrandt3.grid.sara.nl -p5001 -w 2.8M -t 12000 -i 30
------------------------------------------------------------
Client connecting to rembrandt3.grid.sara.nl, TCP port 5001
TCP window size: 5.60 MByte (WARNING: requested 2.80 MByte)
------------------------------------------------------------
[  3] local 128.142.224.234 port 33262 connected with 145.100.3.85 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-30.0 sec  1.49 GBytes   426 Mbits/sec
[  3] 30.0-60.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 60.0-90.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 90.0-120.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 120.0-150.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 150.0-180.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 180.0-210.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 210.0-240.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 240.0-270.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 270.0-300.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 300.0-330.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 330.0-360.0 sec  1.38 MBytes   387 Kbits/sec
[  3] 360.0-390.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 390.0-420.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 420.0-450.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 450.0-480.0 sec  1.02 GBytes   291 Mbits/sec
[  3] 480.0-510.0 sec  2.86 GBytes   819 Mbits/sec
[  3] 510.0-540.0 sec  3.02 GBytes   864 Mbits/sec
[  3] 540.0-570.0 sec  2.92 GBytes   835 Mbits/sec
[  3] 570.0-600.0 sec  3.09 GBytes   885 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3] 600.0-630.0 sec  2.96 GBytes   848 Mbits/sec
[  3] 630.0-660.0 sec  2.93 GBytes   840 Mbits/sec
[  3] 660.0-690.0 sec  2.55 GBytes   729 Mbits/sec
[  3] 690.0-720.0 sec  2.97 GBytes   851 Mbits/sec
[  3] 720.0-750.0 sec  3.06 GBytes   875 Mbits/sec
[  3] 750.0-780.0 sec  3.02 GBytes   864 Mbits/sec
[  3] 780.0-810.0 sec  2.94 GBytes   842 Mbits/sec
[  3] 810.0-840.0 sec  2.85 GBytes   815 Mbits/sec
[  3] 840.0-870.0 sec  3.06 GBytes   876 Mbits/sec
[  3] 870.0-900.0 sec  3.22 GBytes   921 Mbits/sec
[  3] 900.0-930.0 sec  3.22 GBytes   921 Mbits/sec
[  3] 930.0-960.0 sec  2.89 GBytes   828 Mbits/sec
[  3] 960.0-990.0 sec  2.94 GBytes   842 Mbits/sec
<SNIP IPERF REPORT A STABLE THROUGHPUT AROUND 880 Mbits/sec>
[  3] 11760.0-11790.0 sec  3.17 GBytes   909 Mbits/sec
[  3] 11790.0-11820.0 sec  3.09 GBytes   885 Mbits/sec
[  3] 11820.0-11850.0 sec  3.04 GBytes   870 Mbits/sec
[  3] 11850.0-11880.0 sec  2.94 GBytes   841 Mbits/sec
[  3] 11880.0-11910.0 sec  3.03 GBytes   868 Mbits/sec
[  3] 11910.0-11940.0 sec  3.08 GBytes   881 Mbits/sec
[  3] 11940.0-11970.0 sec  3.08 GBytes   882 Mbits/sec
[  3] 11970.0-12000.0 sec  3.13 GBytes   895 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-12000.0 sec  1257321537536 bits   838 Mbits/sec

The connection takes longer to happen with BNL:

[root@lxshare220d ~]# iperf -c gridui01.usatlas.bnl.gov -p21212 -t 12000 -i 30 - w 8M
------------------------------------------------------------
Client connecting to gridui01.usatlas.bnl.gov, TCP port 21212
TCP window size: 16.0 MByte (WARNING: requested 8.00 MByte)
------------------------------------------------------------
[  3] local 128.142.224.220 port 32773 connected with 130.199.185.10 port 21212
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-30.0 sec  23.7 MBytes  6.62 Mbits/sec
[  3] 30.0-60.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 60.0-90.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 90.0-120.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 120.0-150.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 150.0-180.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 180.0-210.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 210.0-240.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 240.0-270.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 270.0-300.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 300.0-330.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 330.0-360.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 360.0-390.0 sec  4.01 MBytes  1.12 Mbits/sec
[  3] 390.0-420.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 420.0-450.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 450.0-480.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 480.0-510.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 510.0-540.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 540.0-570.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 570.0-600.0 sec  0.00 Bytes  0.00 bits/sec
[ ID] Interval       Transfer     Bandwidth
[  3] 600.0-630.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 630.0-660.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 660.0-690.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 690.0-720.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 720.0-750.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 750.0-780.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 780.0-810.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 810.0-840.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 840.0-870.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 870.0-900.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 900.0-930.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 930.0-960.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 960.0-990.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 990.0-1020.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 1020.0-1050.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 1050.0-1080.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 1080.0-1110.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 1110.0-1140.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 1140.0-1170.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 1170.0-1200.0 sec  0.00 Bytes  0.00 bits/sec
[ ID] Interval       Transfer     Bandwidth
[  3] 1200.0-1230.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 1230.0-1260.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 1260.0-1290.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 1290.0-1320.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 1320.0-1350.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 1350.0-1380.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 1380.0-1410.0 sec  0.00 Bytes  0.00 bits/sec
<SNIP, IPERF STILL REPORT 0.00 Bytes sent>
[  3] 3510.0-3540.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 3540.0-3570.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 3570.0-3600.0 sec  0.00 Bytes  0.00 bits/sec
[ ID] Interval       Transfer     Bandwidth
[  3] 3600.0-3630.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 3630.0-3660.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 3660.0-3690.0 sec  41.7 MBytes  11.7 Mbits/sec
[  3] 3690.0-3720.0 sec   406 MBytes   114 Mbits/sec
[  3] 3720.0-3750.0 sec   750 MBytes   210 Mbits/sec
[  3] 3750.0-3780.0 sec   782 MBytes   219 Mbits/sec
[  3] 3780.0-3810.0 sec   785 MBytes   220 Mbits/sec
[  3] 3810.0-3840.0 sec   781 MBytes   218 Mbits/sec
[  3] 3840.0-3870.0 sec   783 MBytes   219 Mbits/sec
[  3] 3870.0-3900.0 sec   781 MBytes   219 Mbits/sec
[  3] 3900.0-3930.0 sec   783 MBytes   219 Mbits/sec
[  3] 3930.0-3960.0 sec   763 MBytes   213 Mbits/sec
<SNIP IPERF REPORT A STABLE THROUGHPUT AROUND 219 Mbits/sec>
[  3] 11940.0-11970.0 sec   781 MBytes   218 Mbits/sec
[  3] 11970.0-12000.0 sec   782 MBytes   219 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-12000.4 sec   209 GBytes   149 Mbits/sec

However the problem does not happen with small buffer/when throughput is not optimal:

[root@lxshare025d ~]# iperf  -c rembrandt3.grid.sara.nl -p5001 -w 1M -t 12000 -i 30
------------------------------------------------------------
Client connecting to rembrandt3.grid.sara.nl, TCP port 5001
TCP window size: 2.00 MByte (WARNING: requested 1.00 MByte)
------------------------------------------------------------
[  3] local 128.142.224.234 port 33260 connected with 145.100.3.85 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-30.0 sec   945 MBytes   264 Mbits/sec
[  3] 30.0-60.0 sec  1.29 GBytes   369 Mbits/sec
[  3] 60.0-90.0 sec  1.23 GBytes   352 Mbits/sec
[  3] 90.0-120.0 sec  1.41 GBytes   404 Mbits/sec
[  3] 120.0-150.0 sec  1.49 GBytes   427 Mbits/sec
[  3] 150.0-180.0 sec  1.41 GBytes   402 Mbits/sec
[  3] 180.0-210.0 sec  1.44 GBytes   411 Mbits/sec
[  3] 210.0-240.0 sec  1.33 GBytes   381 Mbits/sec
[  3] 240.0-270.0 sec  1.44 GBytes   413 Mbits/sec
[  3] 270.0-300.0 sec  1.46 GBytes   417 Mbits/sec
[  3] 300.0-330.0 sec  1.49 GBytes   426 Mbits/sec
[  3] 330.0-360.0 sec  1.47 GBytes   420 Mbits/sec
[  3] 360.0-390.0 sec  1.43 GBytes   409 Mbits/sec
[  3] 390.0-420.0 sec  1.55 GBytes   444 Mbits/sec
[  3] 420.0-450.0 sec  1.37 GBytes   392 Mbits/sec
[  3] 450.0-480.0 sec  1.39 GBytes   399 Mbits/sec
[  3] 480.0-510.0 sec  1.41 GBytes   405 Mbits/sec
[  3] 510.0-540.0 sec  1.37 GBytes   392 Mbits/sec

9th November 2005

10:00

Lots of tests have been performed on the LAN: for some reasons it became hard to replicate the problem in the LAN.

I have installed the latest e1000 drivers (6.2.15-NAPI) and I firstly thought they solved the problem: I could run stable tests with BNL for 30 mins with a very good throughput. After this, during the day the problem came back.

Finally, an interesting behaviour of the system. After a clean reboot, I had the following results:

iperf -c gridui01.usatlas.bnl.gov -p21212 -t 12000 -i 30 -w 8M| tee freshfromreboot.txt
------------------------------------------------------------
Client connecting to gridui01.usatlas.bnl.gov, TCP port 21212
TCP window size: 16.0 MByte (WARNING: requested 8.00 MByte)
------------------------------------------------------------
[  3] local 128.142.224.220 port 32770 connected with 130.199.185.10 port 21212
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-30.0 sec  34.5 MBytes  9.63 Mbits/sec
[  3] 30.0-60.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 60.0-90.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 90.0-120.0 sec  4.01 MBytes  1.12 Mbits/sec
[  3] 120.0-150.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 150.0-180.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 180.0-210.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 210.0-240.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 240.0-270.0 sec  0.00 Bytes  0.00 bits/sec
<SNIP, IPERF STILL REPORT 0.00 Bytes sent>
[  3] 1380.0-1410.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 1410.0-1440.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 1440.0-1470.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 1470.0-1500.0 sec  0.00 Bytes  0.00 bits/sec
<SNIP, IPERF STILL REPORT 0.00 Bytes sent>
[  3] 3000.0-3030.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 3030.0-3060.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 3060.0-3090.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 3090.0-3120.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 3120.0-3150.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 3150.0-3180.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 3180.0-3210.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 3210.0-3240.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 3240.0-3270.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 3270.0-3300.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 3300.0-3330.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 3330.0-3360.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 3360.0-3390.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 3390.0-3420.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 3420.0-3450.0 sec   366 MBytes   102 Mbits/sec
[  3] 3450.0-3480.0 sec   781 MBytes   218 Mbits/sec
[  3] 3480.0-3510.0 sec   781 MBytes   218 Mbits/sec
[  3] 3510.0-3540.0 sec   784 MBytes   219 Mbits/sec
[  3] 3540.0-3570.0 sec   777 MBytes   217 Mbits/sec
[  3] 3570.0-3600.0 sec   781 MBytes   218 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3] 3600.0-3630.0 sec   782 MBytes   219 Mbits/sec
[  3] 3630.0-3660.0 sec   785 MBytes   219 Mbits/sec
[  3] 3660.0-3690.0 sec   781 MBytes   218 Mbits/sec
[  3] 3690.0-3720.0 sec   779 MBytes   218 Mbits/sec
[  3] 3720.0-3750.0 sec   785 MBytes   220 Mbits/sec
[  3] 3750.0-3780.0 sec   686 MBytes   192 Mbits/sec
[  3] 3780.0-3810.0 sec   761 MBytes   213 Mbits/sec
[  3] 3810.0-3840.0 sec   782 MBytes   219 Mbits/sec
[  3] 3840.0-3870.0 sec   716 MBytes   200 Mbits/sec
[  3] 3870.0-3900.0 sec   779 MBytes   218 Mbits/sec
[  3] 3900.0-3930.0 sec   785 MBytes   220 Mbits/sec
[  3] 3930.0-3960.0 sec   782 MBytes   219 Mbits/sec
[  3] 3960.0-3990.0 sec   780 MBytes   218 Mbits/sec
[  3] 3990.0-4020.0 sec   782 MBytes   219 Mbits/sec
[  3] 4020.0-4050.0 sec   783 MBytes   219 Mbits/sec
[  3] 4050.0-4080.0 sec   782 MBytes   219 Mbits/sec
<SNIP IPERF REPORT A STABLE THROUGHPUT AROUND 219 Mbits/sec>
[  3] 11880.0-11910.0 sec   782 MBytes   219 Mbits/sec
[  3] 11910.0-11940.0 sec   784 MBytes   219 Mbits/sec
[  3] 11940.0-11970.0 sec   776 MBytes   217 Mbits/sec
[  3] 11970.0-12000.0 sec   671 MBytes   188 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-12000.3 sec   194 GBytes   139 Mbits/sec

Nothing changed on the system during the test. The following kernel parameters were in place:

net.ipv4.tcp_rmem = 4096 65536 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
net.ipv4.tcp_mem = 98304 131072 134217728
net.core.rmem_max = 134217728
net.core.rmem_default = 87380
net.core.wmem_max = 134217728
net.core.wmem_default = 87380
net.core.optmem_max = 100000
net.core.netdev_max_backlog = 3000
net.ipv4.tcp_sack=0
net.ipv4.tcp_bic = 1
net.ipv4.tcp_vegas_cong_avoid = 0
net.ipv4.tcp_westwood = 0
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0

I am trying to replicate this behaviour.

4th November 2005

10:00

The problem is now a bit harder to replicate on the LAN. A few tests have been run with CNAF (most stable link so far), but the CNAF server stoped responding over the weekend and is now reserved by CNAF for other tests.

3rd November 2005

11:00

It seems very difficult to have another OS on the boxes. Instead, I am trying to reduce the parameter space to isolate a few key factors in the problem.

1st November 2005

16:45

Tests on lxshare025d revealed massive connection losses. In other word the problem appears to be there with a different (Intel) NIC.

I will try to have another 2.6 kernel-based OS running on it, but the system does not have any CD drive and does not boot from USB.

28th October 2005

16:45

Tests from lxshare216d (SLC3) are completed.

lxshare025d has been made available for testing. It is running SLC4, but has a different NIC.

The box was unable to boot properly:

  • selinux had to be disabled
  • notd and notd_exec had to be disabled

The box will run as much tests as possible over the weekend.

24th October 2005

lxshare216d (SLC3. 2.4 kernel) has been allocated for temporary tests. 450+ tests have been performed on 4 sites during the weekend on lxshare216d. Only 3 of them reported a lost connection.

It seems clear to me that the problem we have seen with the network card on 2.6 kernel does not happens with 2.4 kernel.

I think it would be now interesting to do some more testing on 2.6 kernels with another network card.

20th October 2005

16:30

Thanks to the great help of linux support guys, we discovered the following:

- the problem is reproductible on the LAN - the problem is reproductible on the default SLC4 kernel and on a kernel 2.6.13 from kernel.org - the problem is reproductible on the two internal NIC (same chipset)

However: - No problem at all has been detected when replacing lxshare220d by another system.

Conclusion: it looks like the network card is the problem

13:30

Overnight and morning tests revealed that the dropped connections problem is still there with the other NIC. Furthermore, it is replicable in a similar way between CERN and SARA.

Chasing two leads: -> router/switch issue with Edoardo -> Linux kernel/driver issue with Peter

19th October 2005

17:30

lxshare220d is now using the other internal NIC.

17:26

Connectivity issues with SARA solved. Firewall was in the way as with ASCC.

8:54

Connectivity issues solved with Taiwan. Firewall was in the way for some reason.

18th October 2005

11:10

More details about how the connections are dropped: When iperf looks frozen, netstat reports:

Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0 14493420 lxshare220d.cern.ch:32774   192.135.23.200:40000        ESTABLISHED 4101/iperf
Same command, a few seconds later (important value: Send-Q):
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0 14232080 lxshare220d.cern.ch:32774   192.135.23.200:40000        ESTABLISHED 4101/iperf

If iperf is interrupted, network connection goes into:

Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0 14033521 lxshare220d.cern.ch:32774   192.135.23.200:40000        FIN_WAIT1   -

Connections seem to remain in FIN_WAIT1 for a long time. Send-Q seem to be slowly decreasing.

09:30

  • Update from Edoardo and Mark about the connection issue to SARA, they are checking potential firewall issues.

  • Connections drops, as in the following test with one stream, here, seem to happen to one stream, especially at very high rate (> 930Mbits/s). The incident is replicable quite often by adding a parallel UDP stream on the same host. Further testing in progress.

17th October 2005

13:09

Looks like the routing table is incorrect on rembrandt3:

# telnet rembrandt3.grid.sara.nl 5001
Trying 145.100.3.85...
telnet: connect to address 145.100.3.85: No route to host

# traceroute rembrandt3.grid.sara.nl
traceroute to rembrandt3.grid.sara.nl (145.100.3.85), 30 hops max, 38 byte packets
 1  cernh7-sc3 (128.142.224.7)  1.659 ms  0.213 ms  0.294 ms
 2  switch-bckp.ch1.ch.geant.net (62.40.103.181)  0.496 ms  0.380 ms  0.429 ms
 3  ch.fr1.fr.geant.net (62.40.96.30)  8.620 ms  8.644 ms  8.558 ms
 4  fr.uk1.uk.geant.net (62.40.96.90)  15.504 ms  15.504 ms  15.474 ms
 5  uk.nl1.nl.geant.net (62.40.96.181)  22.972 ms  22.874 ms  22.914 ms
 6  surfnet-gw.nl1.nl.geant.net (62.40.103.98)  23.975 ms  23.079 ms  23.348 ms
 7  po11-0.cr1.amsterdam1.surf.net (145.145.166.33)  25.471 ms  23.076 ms  23.098 ms
 8  PO0-0.AR5.Amsterdam1.surf.net (145.145.162.2)  224.601 ms  23.575 ms  240.087 ms
 9  Gi1-2-1.XSR01.Amsterdam1A.surf.net (145.145.166.22)  23.597 ms  23.825 ms  23.721 ms
10  vl1068.sara-r1.sara.nl (145.145.19.98)  23.599 ms  23.327 ms  23.345 ms
11  vl1068.sara-r1.sara.nl (145.145.19.98)  23.469 ms !X *  23.518 ms !X

12:33

iperf results are available here for connections between CERN and:

  • CNAF
  • BNL
  • ASGC

-- Main.rwartel - 17 Oct 2005

Edit | Attach | Watch | Print version | History: r20 < r19 < r18 < r17 < r16 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r20 - 2005-12-01 - RomainWartel
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback