+1 for massive packet reordering in simpe IPSEC+GRE case.
Bug can be easily reproduced on 2 CCRs connected by typical 5..15Mb WAN.
iperf3 in UDP mode reporting out of order packets even on manual bandwidth regulation (10-30% of actual link bandwidth)
[root@gw-sev.bigcar.local ~]# iperf3 -c 192.168.2.2 -b 1M -u -V iperf 3.0.11 Linux gw-sev.bigcar.local 2.6.32-573.12.1.el6.x86_64 #1 SMP Tue Dec 15 21:19:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux Time: Fri, 25 Dec 2015 21:51:28 GMT Connecting to host 192.168.2.2, port 5201 Cookie: gw-sev.bigcar.local.1451080288.34838 [ 4] local 10.1.0.7 port 45391 connected to 192.168.2.2 port 5201 Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 10 second test [ ID] Interval Transfer Bandwidth Total Datagrams [ 4] 0.00-1.00 sec 112 KBytes 917 Kbits/sec 14 [ 4] 1.00-2.00 sec 120 KBytes 983 Kbits/sec 15 [ 4] 2.00-3.00 sec 128 KBytes 1.05 Mbits/sec 16 [ 4] 3.00-4.00 sec 120 KBytes 983 Kbits/sec 15 [ 4] 4.00-5.00 sec 120 KBytes 983 Kbits/sec 15 [ 4] 5.00-6.00 sec 128 KBytes 1.05 Mbits/sec 16 [ 4] 6.00-7.00 sec 120 KBytes 983 Kbits/sec 15 [ 4] 7.00-8.00 sec 120 KBytes 983 Kbits/sec 15 [ 4] 8.00-9.00 sec 120 KBytes 983 Kbits/sec 15 [ 4] 9.00-10.00 sec 128 KBytes 1.05 Mbits/sec 16 - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 0.00-10.00 sec 1.19 MBytes 996 Kbits/sec 0.833 ms 42/152 (28%) [ 4] Sent 152 datagrams CPU Utilization: local/sender 0.4% (0.0%u/0.4%s), remote/receiver 0.0% (0.0%u/0.0%s)
(iperf client counts reordered packets as lost)
Server listening on 5201 ----------------------------------------------------------- Accepted connection from 10.1.0.7, port 46784 [ 5] local 192.168.2.2 port 5201 connected to 10.1.0.7 port 45391 iperf3: OUT OF ORDER - incoming packet = 3 and received packet = 4 AND SP = 5 iperf3: OUT OF ORDER - incoming packet = 6 and received packet = 7 AND SP = 5 .... iperf3: OUT OF ORDER - incoming packet = 148 and received packet = 149 AND SP = 5 iperf3: OUT OF ORDER - incoming packet = 151 and received packet = 152 AND SP = 5 [ 5] 9.00-10.00 sec 128 KBytes 1.05 Mbits/sec 0.833 ms 5/16 (31%) [ 5] 10.00-10.09 sec 0.00 Bytes 0.00 bits/sec 0.833 ms 0/0 (-nan%) - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-10.09 sec 1.19 MBytes 987 Kbits/sec 0.833 ms 42/152 (28%) [SUM] 0.0-10.1 sec 42 datagrams received out-of-order ----------------------------------------------------------- Server listening on 5201 -----------------------------------------------------------
And this is only 1Mbit between 2 CCRs.
IPSec is working stable - UDP flooding over IPSEC always utilise all bandwidth. Statistics is almost clean:
[artem@mow01.r.severscan.ru] > /ip ipsec statistics print in-errors: 0 in-buffer-errors: 0 in-header-errors: 0 in-no-states: 60 in-state-protocol-errors: 416 in-state-mode-errors: 0 in-state-sequence-errors: 0 in-state-expired: 0 in-state-mismatches: 0 in-state-invalid: 49 in-template-mismatches: 0 in-no-policies: 0 in-policy-blocked: 0 in-policy-errors: 0 out-errors: 0 out-bundle-errors: 0 out-bundle-check-errors: 0 out-no-states: 594195 out-state-protocol-errors: 4242 out-state-mode-errors: 0 out-state-sequence-errors: 0 out-state-expired: 4242 out-policy-blocked: 0 out-policy-dead: 0 out-policy-errors: 0
All routerboards are flashed with latest firmware
革命制度党(artem@noz01.r.severscan.ru) > /系统包nt Flags: X - disabled # NAME VERSION SCHEDULED 0 routeros-tile 6.33 1 system 6.33 2 X wireless-cm2 6.33 3 X ipv6 6.33 4 wireless-fp 6.33 5 hotspot 6.33 6 dhcp 6.33 7 mpls 6.33 8 routing 6.33 9 ppp 6.33 10 security 6.33 11 advanced-tools 6.33 [artem@noz01.r.severscan.ru] > /system routerboard print routerboard: yes model: CCR1009-8G-1S serial-number: ************* firmware-type: tilegx current-firmware: 3.27 upgrade-firmware: 3.27