I have an odd situation in that two virtual RouterOS instances, both running 6.49.4 (x86), support IPSec AES hardware offloading differently. Any ideas on where I could hunt?
Both have default IPSec settings:
[user@router] > ip ipsec export verbose # feb/18/2022 12:44:25 by RouterOS 6.49.3 # software id = xxxx-xxxx # # # /ip ipsec mode-config set [ find default=yes ] name=request-only responder=no use-responder-dns=exclusively /ip ipsec policy group set [ find default=yes ] name=default /ip ipsec profile set [ find default=yes ] dh-group=modp2048,modp1024 dpd-interval=2m dpd-maximum-failures=5 \ enc-algorithm=aes-128,3des hash-algorithm=sha1 lifetime=1d name=default nat-traversal=yes proposal-check=obey /ip ipsec proposal set [ find default=yes ] auth-algorithms=sha1 disabled=no enc-algorithms=aes-256-cbc,\ aes-192-cbc,aes-128-cbc lifetime=30m name=default pfs-group=modp1024 /ip ipsec policy set 0 disabled=no dst-address=::/0 group=default proposal=default protocol=all src-address=::/0 template=yes /ip ipsec settings set accounting=yes interim-update=0s xauth-use-radius=no [user@router] > ip ipsec export # feb/18/2022 12:44:28 by RouterOS 6.49.3 # software id = xxxx-xxxx # # #
L2TP IPSec configuration is identical on the two routers:
/ppp profile set *FFFFFFFE only-one=yes /ppp secret add local-address=192.168.253.5 remote-address=192.168.253.6 service=l2tp name=site password=**************** /interface l2tp-server server set authentication=mschap2 enabled=yes ipsec-secret=******************************** \ max-mru=1379 max-mtu=1379 use-ipsec=required /interface l2tp-server add name=vpn-site user=site
The two virtual routers have identical KVM virtualisation containers, with the same CPU family and feature codes exposed. Herewith the CPU information for when we start the virtual container using Debian 11 (bullseye):
(root@debian11 ~) #猫/proc/cpuinfo处理器:0vendor_id : GenuineIntel cpu family : 6 model : 61 model name : Intel Core Processor (Broadwell) stepping : 2 microcode : 0x1 cpu MHz : 2394.454 cache size : 16384 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit srbds bogomips : 4788.90 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 61 model name : Intel Core Processor (Broadwell) stepping : 2 microcode : 0x1 cpu MHz : 2394.454 cache size : 16384 KB physical id : 1 siblings : 1 core id : 0 cpu cores : 1 apicid : 1 initial apicid : 1 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit srbds bogomips : 4788.90 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management:
Detected system resources in RouterOS are identical between the two VMs:
[user@router] > sys resource print uptime: 6h18m11s version: 6.49.3 (stable) build-time: Dec/22/2021 13:49:22 free-memory: 202.3MiB total-memory: 256.0MiB cpu: Intel cpu-count: 2 cpu-frequency: 2194MHz cpu-load: 5% free-hdd-space: 67.3MiB total-hdd-space: 95.3MiB write-sect-since-reboot: 85096 write-sect-total: 85096 architecture-name: x86 board-name: x86 platform: MikroTik
Working router (shows hw-aead as being active 'HE'):
[user@router] > / ip ipsec installed-sa打印的地方state=mature Flags: H - hw-aead, A - AH, E - ESP 0 HE spi=0xED7D7AE src-address=1.1.103.49:4500 dst-address=2.2.72.16:4500 state=mature auth-algorithm=sha1 enc-algorithm=aes-cbc enc-key-size=256 auth-key="8a23a02449d3a1edd01d24b93ce0a85a84c1a0b1" enc-key="cc69d8ea1b08fa311a1d5616e3804731df2a1a8afefd52e40512f0cf82cfa8f4" addtime=feb/18/2022 12:38:55 expires-in=15m40s add-lifetime=24m/30m current-bytes=92354 current-packets=912 replay=128 1 HE spi=0xDE4CD14 src-address=2.2.72.16:4500 dst-address=1.1.103.49:4500 state=mature auth-algorithm=sha1 enc-algorithm=aes-cbc enc-key-size=256 auth-key="489dd6dc11b292c08a3a0dc4bc761da3c38cc972" enc-key="6cc009d1dcddd980bb1d6bcb402be5d47a35d489e25a4bad0e1634b89cf85a18" addtime=feb/18/2022 12:38:55 expires-in=15m40s add-lifetime=24m/30m current-bytes=92086 current-packets=912 replay=128
Non hardware accelerated router (shows hw-aead being inactive 'E'):
[user@router] > / ip ipsec installed-sa打印的地方state=mature Flags: H - hw-aead, A - AH, E - ESP 0 E spi=0x6E05009 src-address=1.1.35.120 dst-address=2.2.23.89 state=mature auth-algorithm=sha1 enc-algorithm=aes-cbc enc-key-size=256 auth-key="8b10c0e4fd943fc598295d0ab1540b356e5ecefd" enc-key="e5dcd3cdf09988ed298cf50acf2b250df040ed0b09d48cf0788df889d5a0079c" addtime=feb/18/2022 12:31:00 expires-in=7m41s add-lifetime=24m/30m current-bytes=425537 current-packets=2947 replay=128 1 E spi=0x12C8896 src-address=2.2.23.89 dst-address=1.1.35.120 state=mature auth-algorithm=sha1 enc-algorithm=aes-cbc enc-key-size=256 auth-key="1370d7011235de40722b1227ff46d0434ebd9f33" enc-key="d09001b3f2d6c83e85a2bf01ce3fa6e7578981d2d80d70f9f997207537be056a" addtime=feb/18/2022 12:31:00 expires-in=7m41s add-lifetime=24m/30m current-bytes=158794 current-packets=1554 replay=128
PS: I can further validate that AES-NI is working when booting the VM with Debian 11 (bullseye). The first result is without it using AES-NI, the second is with it enabled:
[root@debian11 ~]# openssl speed aes-256-cbc Doing aes-256 cbc for 3s on 16 size blocks: 22326434 aes-256 cbc's in 2.98s Doing aes-256 cbc for 3s on 64 size blocks: 5773155 aes-256 cbc's in 2.99s Doing aes-256 cbc for 3s on 256 size blocks: 1477639 aes-256 cbc's in 2.99s Doing aes-256 cbc for 3s on 1024 size blocks: 368841 aes-256 cbc's in 2.99s Doing aes-256 cbc for 3s on 8192 size blocks: 45853 aes-256 cbc's in 2.99s Doing aes-256 cbc for 3s on 16384 size blocks: 23164 aes-256 cbc's in 2.99s OpenSSL 1.1.1k 25 Mar 2021 built on: Tue Aug 24 08:28:12 2021 UTC options:bn(64,64) rc4(16x,int) des(int) aes(partial) blowfish(ptr) compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/openssl-MBf3dh/openssl-1.1.1k=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2 The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes aes-256 cbc 119873.47k 123572.55k 126513.57k 126318.79k 125628.02k 126929.42k [root@debian11 ~]# openssl speed -evp aes-256-cbc Doing aes-256-cbc for 3s on 16 size blocks: 65818605 aes-256-cbc's in 2.99s Doing aes-256-cbc for 3s on 64 size blocks: 18056419 aes-256-cbc's in 2.99s Doing aes-256-cbc for 3s on 256 size blocks: 4754481 aes-256-cbc's in 2.99s Doing aes-256-cbc for 3s on 1024 size blocks: 1174841 aes-256-cbc's in 2.99s Doing aes-256-cbc for 3s on 8192 size blocks: 148046 aes-256-cbc's in 2.99s Doing aes-256-cbc for 3s on 16384 size blocks: 74660 aes-256-cbc's in 2.98s OpenSSL 1.1.1k 25 Mar 2021 built on: Tue Aug 24 08:28:12 2021 UTC options:bn(64,64) rc4(16x,int) des(int) aes(partial) blowfish(ptr) compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/openssl-MBf3dh/openssl-1.1.1k=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2 The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes aes-256-cbc 352206.58k 386491.91k 407072.62k 402353.57k 405616.33k 410479.68k