Bandwidth with Simple Queue lower than expected (cake)

petertosh · Mon Aug 01, 2022 11:36 am

Hi! I am operating a hotspot with about 400 active users max (17 APs, outdoor). WAN is a fibre line with about 260/60 MBit limit. The hotspot users use the ip-pool 192.168.0.0/23. Internal network is on 192.168.177.0/24. The following queues are used to limit each user to about 15/10 MBit/s. The queues are setup like that:

/queue type add kind=pcq name=pcq-download-k1 pcq-classifier=dst-address pcq-rate=16M add kind=pcq name=pcq-upload-k1 pcq-classifier=src-address pcq-rate=11M add cake-diffserv=besteffort cake-nat=yes kind=cake name=cake-default add cake-flowmode=dual-srchost cake-nat=yes kind=cake name=cake-WAN-tx add cake-diffserv=besteffort cake-flowmode=dual-dsthost cake-nat=yes kind=cake name=cake-WAN-rx /queue simple add bucket-size=0/0 max-limit=65M/255M name=cake-regular queue=cake-WAN-tx/cake-WAN-rx target=192.168.0.0/16 total-queue=cake-default add bucket-size=0/0 max-limit=65M/255M name=cake-child parent=cake-regular queue=cake-WAN-tx/cake-WAN-rx target=192.168.0.0/16 \ total-queue=cake-default add bucket-size=0/0 max-limit=40M/230M name=Hotspot-Night \ parent=cake-regular queue=pcq-upload-default/pcq-download-default target=192.168.0.0/23 time=23h30m-7h,sun,mon,tue,wed,thu,fri,sat \ total-queue=hotspot-default add bucket-size=0/0 max-limit=40M/230M name=Hotspot-Day \ parent=cake-regular queue=pcq-upload-k1/pcq-download-k1 target=192.168.0.0/23 time=7h-23h30m,sun,mon,tue,wed,thu,fri,sat total-queue=\ hotspot-default

Now in the evening hours the total bandwidth going out to the hotspot is somehow limited to about 150 MBit/s download. Attached is a screenshot from dude. ROS is 7.3.1 on a CCR1009.

Why is the download bandwidth not higher? What have I done wrong with my queue setup?

petertosh · Fri Aug 05, 2022 12:27 am

After playing around with my homelab and later with the hotspot-router, I found a solution, although I don't understand what exactly the changes do.

With this setup I am both able to control the latency of the WAN (both internal and hotspot, waveform bufferbloat test results in A and A+) and reach the expected bandwith limits:

/queue simple add bucket-size=0.1/0.01 max-limit=65M/230M name=cake-regular queue=cake-WAN-tx/cake-WAN-rx target=192.168.0.0/16 add bucket-size=0.1/0.01 max-limit=40M/220M name=Hotspot-Day parent=cake-regular queue=pcq-upload-k1/pcq-download-k1 target=192.168.0.0/23 time=7h-23h30m,sun,mon,tue,wed,thu,fri,sat total-queue=hotspot-default add bucket-size=0.1/0.01 max-limit=40M/220M name=Hotspot-Night parent=cake-regular queue=pcq-upload-default/pcq-download-default target=192.168.0.0/23 time=23h30m-7h,sun,mon,tue,wed,thu,fri,sat total-queue=hotspot-default add bucket-size=0.1/0.01 name=cake-child parent=cake-regular queue=cake-WAN-tx/cake-WAN-rx target=192.168.0.0/16 /queue type add kind=pcq name=pcq-download-k1 pcq-classifier=dst-address pcq-rate=16M add kind=pcq name=pcq-upload-k1 pcq-classifier=src-address pcq-rate=11M add cake-flowmode=dual-srchost cake-nat=yes kind=cake name=cake-WAN-tx add cake-diffserv=besteffort cake-flowmode=dual-dsthost cake-nat=yes kind=cake name=cake-WAN-rx

The difference between this and my previous settings is basically that the "Total Queue Type" of the main and the child queue is now "default-small" instead of cake. Why does this change the throughput so drastically?

In general the result looks good to me. The flent test ran in parallel to the hotspot traffic (on a wired raspberrypi on the internal network), therefore upload and download are varying.

dtaht · Sun Aug 14, 2022 9:05 pm

Why do you feel it is necessary to also rate limit each ap? cake does per host fq itself (although, if natting, you need the nat option on)

Certainly it is helpful to run most wifi APs at well below their maximum rate unless they have this:https://www.cs.kau.se/tohojo/airtime-fairness/

An easy way to verify if cake is actually working is to use diffserv3 or diffserv4 rather than besteffort, against the rrul (not rrul_be) test. In this scenario and load besteffort is probably best.

No idea what the small queues thing does. Elsewhere I've been agitating that the tx queue (ring buffer) size be vastly reduced by default on mikrotik wifi.

petertosh · Mon Aug 15, 2022 11:47 pm

Thank you very much for your work and insights on this forum! I limit each wireless client in order to achieve something like a manual airtime fairness. I guess my APs don't have airtime fairness as this option was removed from the controller some time ago. So I thought, in limiting each client, the AP will have spare airtime to handle all requests. Do you think I could remove this limit and leave it to cake? But cake runs on the router, not on the APs, so it can't take care of a congested AP?

Here come three rrul tests I did just now. The network/hotspot is humming right now, so the throughput is varying. When changing from besteffort to diffserv4 it seemed the hotspot throughput was lowered (so my flent-machine had more bandwidth). Putting it back on besteffort made it recover.

petertosh · Tue Aug 16, 2022 12:41 am

相比之下,rrul测试用蛋糕和简单的,ue disabled. I guess cake is doing its magic (in the tests above).

Bandwidth with Simple Queue lower than expected (cake)

Bandwidth with Simple Queue lower than expected (cake)

Re: Bandwidth with Simple Queue lower than expected

Re: Bandwidth with Simple Queue lower than expected (cake)

Re: Bandwidth with Simple Queue lower than expected (cake)

Re: Bandwidth with Simple Queue lower than expected (cake)

Who is online