qos

Omada gateway QoS is a cap, not a priority — what actually works

TL;DR: On a TP-Link Omada gateway (ER707-M2, controller 6.2.14), the Gateway QoS
class percentages are hard caps. A class never goes above its share, even when the line is
idle. To get “bulk downloads use everything, but back off when something important
starts”, use Bandwidth Control with Threshold Control, and get three numbers right:

  1. Declare the speed your line really delivers, not the speed on your contract. The threshold
    is a percentage of the number you type in. If that number is too high, the cap never engages.
  2. Limit bulk at the source to below the trigger, so bulk alone never trips its own cap.
  3. Make the cap small enough that backing off really frees the line: 250 Mbps on a 930 Mbps line, not 500.

Every number below comes from a measured test, not the docs.

The goal

I have a VLAN for bulk traffic (torrents, model downloads, backups) and other VLANs
for everything else: IoT and household devices, a low-latency gaming network, and the
default LAN. The main WAN is a ~1 Gbps fiber line, and a second slower line is a backup.

What I wanted is simple to say:

Bulk gets all the bandwidth nobody else is using. The moment something more
important starts, bulk shrinks.

On a Linux router that’s an afternoon with CAKE or fq_codel. On Omada, it isn’t straightforward.

How I tested

  • One throwaway container per VLAN (an Alpine LXC on Proxmox, tagged for the VLAN), so the traffic
    really comes from that network, through the switch and the gateway.
  • 8 parallel downloads from two Singapore speed-test servers per container. I counted
    received bytes every 2 seconds from /sys/class/net/eth0/statistics/rx_bytes, so I could see
    the shape over time, not just an average. (A single averaged number would have hidden
    most of what’s below.) Later tests ran for minutes and read each container’s counters from the
    Proxmox host, so both sides were counted separately.
  • Four scenarios for every setting: bulk alone, important traffic alone, both starting
    together, and the realistic one, important traffic joining while bulk is already running.
  • Every change went through a small approval box I run. It snapshots the controller VM,
    applies the change, watches the internet for a few minutes, and rolls back automatically
    if the internet drops. That made it cheap to try things that might be wrong.

My test script (run inside each VLAN container):

#!/bin/sh
# dl2.sh SECONDS STREAMS -> download Mbps every 2 s, then the average
T=${1:-20}; N=${2:-8}
i=0; while [ $i -lt $N ]; do
  if [ $((i%2)) -eq 0 ]; then u=https://sin-speed.hetzner.com/10GB.bin
  else u=http://speedtest.singapore.linode.com/1GB-10M.bin; fi
  curl -s -o /dev/null -m $T "$u" &
  i=$((i+1)); done
f=/sys/class/net/eth0/statistics/rx_bytes; start=$(cat $f); prev=$start; s=""; k=0
while [ $k -lt $((T/2)) ]; do sleep 2; cur=$(cat $f)
  s="$s $(( (cur-prev)*8/2/1000000 ))"; prev=$cur; k=$((k+1)); done
wait
echo "avg $(( (prev-start)*8/T/1000000 )) Mbps | every 2s:$s"


Pick a server close to you, and check your speed-test box first. My self-hosted speed tracker
reported 340–420 Mbps on a line that really does 850–950. The box was the limit, not
the line.

Baseline: no rules

ScenarioBulk VLANIoT VLAN
Alone810 Mbps858 Mbps
Both start together671245

Bulk wins because it has more connections open. That’s the problem.

Myth 1: the per-network “QoS” setting

Every LAN network in Omada has a QoS / priority field. The gateway doesn’t use it for WAN
shaping.
It’s an 802.1p tag that switches and access points use to queue traffic locally. Setting
bulk to 0 and IoT to 4 changed nothing on the internet link.

Myth 2: Gateway QoS class ratios are guaranteed minimums

Settings → Transmission → QoS has one bandwidth rule per WAN, with four ratios
(Class 1, 2, 3, Others; they add up to 100), and class rules that map source IP groups to classes. I read
the ratios as guaranteed minimums that lend unused bandwidth to the other classes, which is how most
ratio-based QoS works. TP-Link’s guide never says either way, but its own example gives
Guest 2% and IoT 1%, which only makes sense as caps.

I set the WAN to 975 Mbps, with IoT = Class 2 at 45% and bulk = Class 3 at 5%:

ScenarioNo rulesClass ratios on
Bulk alone828 Mbps47 (5% of 975 = 49)
IoT alone933434 (45% = 439)
Both499 / 28651 / 382

Each class is capped at its share even when nothing else is using the line. Unused
bandwidth isn’t lent to anyone. I removed it within minutes.

Two smaller findings from this test:
– The WAN speed field in this API is in Kbps, with a maximum of 1,000,000, so you can’t enter
more than 1 Gbps here.
– Class rules do nothing until the WAN has a bandwidth rule.

What works: Bandwidth Control + Threshold Control

Settings → Transmission → Bandwidth Control is a different feature. Rules cap a
network or IP group at a fixed speed. With Threshold Control turned on, the caps apply
only while total WAN usage is above a percentage of the WAN speed you enter. That’s
close to “back off when the line is busy”. Here’s how it actually behaves.

Attempt 1: threshold 80%, bulk capped at 200 Mbps

ScenarioBulkIoT
Bulk running, IoT joins~190~850
Bulk alone, below 80%~620 (not capped)—
Bulk alone, bursting to ~1000800 → 1044 → 1021 → ~190—
Bulk alone right after that~195 for 30 s+—

IoT wins, but two surprises:

  1. Bulk trips the threshold by itself. The threshold counts all WAN traffic, bulk
    included. Bulk alone above 80% capped itself after about 10 seconds.
  2. The cap stays on after it trips. It stayed on for 40+ seconds with the line nearly idle.
    (Later, longer tests showed it can hold for four to five minutes. More on that below.)

So in practice the cap is the floor bulk drops to after any busy moment, not
only while something important is running. At 200 Mbps that wastes most of the line.

Attempt 2: threshold 90%, bulk capped at 600 Mbps

My idea was to give bulk more room before the threshold trips. It was much worse:

ScenarioBulkIoT
Bulk running, IoT joins~590–750 (above its 600 cap)~7–12 Mbps
Bulk on only 2 connections + IoT421434

The threshold is a percentage of the WAN speed you entered (1024), not of what the line
delivers (~850–950).
90% = 922 Mbps, which the line rarely reached, so the rule almost
never switched on. With no cap, an already-running 8-connection download leaves a newly
started download almost nothing. That’s worse than having no rule, because in that run
both had started together.

Keep that entered-versus-real gap in mind. It comes back later.

Attempt 3: threshold 80%, bulk capped at 500 Mbps

ScenarioNo rules80% / 20090% / 60080% / 500
Bulk running, IoT joins—190 / 850~650 / ~101000 → ~480 / → ~440
Both start together671 / 245193 / 410571 / 14476 / 378 → 484
IoT alone858855607–910948
Bulk alone right after a busy moment810~195~590~480

The 2-second samples from the realistic scenario show it best:

bulk: 225 1043 1001 1052 1001 618 437 599 569 474 478 466 516 492 440
iot :                      162 166 272 330 331 372 443 464 400 424


Bulk runs at full line speed, IoT starts, and within seconds bulk drops to its 500 cap
while IoT climbs to fill the rest. I ran this for a few days. It had two problems left:
bulk still burst to ~1000 and capped itself all the time, and 500 of a ~930 line meant
“backing off” still left bulk with half the bandwidth.

Step 4: stop bulk from tripping itself — limit it at the source

The gateway can’t tell “bulk is busy” from “the house is busy”. So I stopped bulk from ever
getting near the trigger on its own. My bulk hosts are Proxmox containers, and a guest’s network
interface takes a rate= limit (in MB/s, so ×8 for Mbps). A download client’s global speed
limit or tc on the host works just as well.

Source limitBulk alone, 8 connections, 80% threshold
none~1000 for ~10 s, then ~480 (tripped its own cap)
680 Mbpstripped too: 680 of bulk plus normal house traffic is over the line
600 Mbps~600 held for 3 minutes, never capped

Now bulk alone runs flat out just under the trigger, and it takes a second user to push
the line over it. That’s the behavior I wanted from the start.

Step 5: the real bug — the speed I declared

Then someone in the house ran a speed test while bulk was downloading and got 408 Mbps. I
checked the rule and it had never engaged.

By then I’d raised the threshold a little, to 83%. With the WAN still declared as 1024
(the contract speed), the trigger sat at 850 Mbps. So I measured the line properly: 6 parallel
streams, from a test container with 4 cores instead of 1. A single core can’t push a gigabit
through TLS, and that’s what had kept my earlier numbers low. The line does 933 down / 891 up.
Real traffic almost never gets above 850 on it:

BulkIoT (6 connections)Total
Bulk alone611 Mbps—611
IoT loading the line at the same time528299827

Bulk gave up 83 Mbps. Nothing was capped, because the total never reached 850. The gateway
never saw the line as busy, and the two just shared it as ordinary TCP flows.

The declared WAN speed is a policy setting, not a label. It decides where every cap starts to
work. I declared what I’d measured (930 / 890) and lowered the bulk cap from 500 to 250.

Here’s the same test after the change, run for 150 seconds with IoT downloading the whole time:

WindowBulkIoTTotal
0–20 s246655900
20–40 s246720966
40–120 s246634–679880–925

IoT went from 299 to ~670 Mbps on the same test. Bulk sat on its cap and IoT took the rest.

Two things this run taught me:

  • A 30-second test is too short. The first run after the change showed bulk 390 / IoT 377.
    That was the ramp, not where it settles. Measure QoS over minutes.
  • The cap releases slowly. After the load stopped, bulk stayed at 246 Mbps for about
    four and a half minutes, then jumped back to ~610 in one step. So a short burst of other
    traffic costs bulk several minutes of speed. For a download box that’s a fair price for the
    line never feeling slow, but know about it before you decide the cap is stuck.

Upload behaves the same way. With bulk and IoT both uploading, bulk sat at 241–244 (its cap)
and IoT got 592–642. One rule covers both directions.

Step 6: tuning the gap — how often does it fire?

The cap was working. Was it working too often? I counted over the ~9 hours the bulk downloader
ran. I compared the bulk container’s traffic with the total on the line, and treated the
difference as “the house”:

Times the cap switched on63 (median 3.5 min each, longest 7.5 min)
Bulk running at its cap39.5% of its active time
House traffic while bulk was capped (median)184 Mbps
House traffic while bulk ran free (median)74 Mbps

So most of the time the cap had a real reason. Only about 5% of bulk’s running time was spent
capped with the house nearly idle, which is the slow release. It switched on so often because
bulk’s own ceiling (~610) sat only 162 Mbps below the trigger (772 = 83% of 930). Ordinary
household use crosses that several times an hour.

I widened the gap from both sides. I lowered bulk’s source limit to ~510 Mbps and raised the
threshold to 86%, which moves the trigger to 800. Now the house can use about 290 Mbps
before bulk has to back off.

Where it ended up

SettingValue
Main WAN declared speed930 / 890 Mbps (measured, not the contract’s 1024)
Threshold Control86% → caps engage above 800 down / 765 up
Bulk VLAN cap on the main WAN250 / 250 Mbps, shared by the whole VLAN
Bulk VLAN cap on the backup WAN150 / 150 Mbps (the backup measured 486 / 752)
Bulk hosts, limited at the source~510 Mbps each

How to set it up

  1. Measure your real line speed, in both directions. Use several parallel connections (one
    connection measures the route to one server, not your line), from a wired host with enough CPU
    to push that much. Let the transfers finish instead of cutting them off.
  2. Declare the measured speed in Bandwidth Control. Don’t use the contract speed.
  3. Choose the threshold so the trigger (declared speed × %) sits clearly above what bulk
    plus normal background traffic uses. It also has to sit below what the line reaches when two
    things are busy at once. I use 86% of 930 = 800.
  4. Limit bulk at the source to well below the trigger, so bulk alone never trips its own cap.
    The gap between the two is how much the rest of the house can use before bulk backs off.
  5. Create one rule per WAN for the bulk network: source = that network, mode = Shared
    (the whole VLAN shares the cap). A rule only covers the WANs you tick.
  6. Make the cap small: the speed bulk drops to when it backs off, and possibly keeps for
    several minutes. Whatever’s left (line − cap) is what everyone else is sure to get.
  7. Test the realistic case over minutes: important traffic starting while bulk is already
    running. That’s where bad settings show up.
  8. Check how often it fires after a day of normal use, and move the gap if it’s too often.

Caveats

  • The cap sticks for minutes after the line was busy (about 4.5 minutes in my longest
    measurement), even if bulk is the only traffic left.
  • The threshold is one percentage for every WAN. You can’t give the backup line its own. On
    a slow backup (or an LTE line you’ve never measured), a wrong declared speed can mean its caps
    never engage.
  • A rule only covers the WANs you tick. If traffic fails over to a backup WAN, bulk
    runs uncapped there unless it has its own rule.
  • Don’t measure from the controller. The gateway’s WAN byte counters in the controller API only
    refresh every 2–3 minutes, so rates worked out from them come out doubled or halved. The same data
    has rxRate / txRate fields (in KiB/s, refreshed about every 30 s) that are usable. Counting
    bytes on the hosts themselves is better still.
  • This is one gateway model and one controller version. Don’t assume other models behave the same.
    Measure them.
  • It’s not smart queueing. It’s a switch that turns on at a traffic level. If you need latency
    control under load (gaming during a big download), the real answer is still CAKE/fq_codel on a
    Linux or BSD router.

What I learned beyond Omada

  • Average numbers lie. Two-second samples showed the cap tripping, staying on and lifting.
    Averages would have made the 200-cap test look “roughly fine” and the 90% test look
    like random noise.
  • But short tests lie too. Thirty seconds showed a ramp and I nearly read it as a result.
    Look closely, and also over a long enough time.
  • Test the assumption that sounds obvious. “Ratio = guaranteed minimum”, “higher
    threshold = more room for bulk” and “the WAN speed field is just the contract speed” all
    sounded right, and all three were wrong.
  • Make changes cheap to undo. Snapshot → apply → watch → auto-rollback turned
    a few dozen risky router changes into something I could run on a weeknight.