HFSC - How to properly borrow.
Richard Wilson
Despite searching both this and the misc@ lists, and porting to misc@
and getting useful responses that I should look at
http://www.probsd.net/pf/index.php/Hednod%27s_HFSC_explained and
http://calomel.org/pf_hfsc.html (which I did) I am still failing
spectacularly to understand how to properly do HFSC.
Having previously posted asking 'How do I do this?' I though perhaps I
would get better answers if I tried 'This is what I've tried, what am I
doing wrong?'.
What I'm trying to do:
We have a leased line for hosting our servers. We don't use anywhere
near that much, so we resell the spare capacity to other tenants in our
building, providing a {A,S}DSL-like service. In our setup, this means
that we have pools of 20 clients, who then contend for the bandwidth
allocated to that pool. ADSL-alike gets 2Mb down and 256Kb up, SDSL gets
2Mb both ways.
At the moment, we are on an old setup, from before I'd heard of HFSC.
There is a single pair of rules for each pool, and all clients in that
pool get their up- and downstream traffic assigned to those queues. This
sucks, because if any one client tries to whore over their connection,
they can take up the entire allocation for their pool.
Instead, I would like to have it so that each client has their own
queue, which is guaranteed their 20th of the bandwidth, and then
contends with the other clients in their pool for the remaining 19
20ths. Below is my initial attempt at the queue definitions:
# Queueing
altq on $ext_if bandwidth 9.7Mb hfsc(linkshare 9.7Mb upperlimit 9.7Mb)
queue { adsl_up, sdsl_up, default_up }
altq on $int_if bandwidth 9.7Mb hfsc(linkshare 9.7Mb upperlimit 9.7Mb)
queue { adsl_dn, sdsl_dn, default_dn }
# All numbers set the same, because that's how the Hednod example had.
Not sure why.
queue adsl_up bandwidth 256Kb hfsc(realtime 128Kb linkshare 256Kb) {
adsl_client1_up, adsl_client2_up }
queue adsl_dn bandwidth 2048Kb hfsc(realtime 512Kb linkshare 2048Kb) {
adsl_client1_dn, adsl_client2_dn }
queue sdsl_up bandwidth 2048Kb hfsc(realtime 512Kb linkshare 2048Kb) {
sdsl_client1_up, sdsl_client2_up }
queue sdsl_dn bandwidth 2048Kb hfsc(realtime 512Kb linkshare 2048Kb) {
sdsl_client1_dn, sdsl_client2_dn }
# (Not sure if the realtime bit is needed...)
# Not sure how to do this bit at all in fact.
# Want it so that each pool can take up to 2Mb, but if they're not using
it, its available to default.
# ADSL
# Clients get a guaranteed 1/20th of the bandwidth, and fight with their
peers
# over the remaining 19/20th.
# 256/20=12, 2048/20=102 (rounded to whole Kb)
queue adsl_client1_up bandwidth 12Kb hfsc(realtime 12Kb linkshare 256Kb)
queue adsl_client1_dn bandwidth 100Kb hfsc(realtime 100Kb linkshare 2048Kb)
queue adsl_client2_up bandwidth 12Kb hfsc(realtime 12Kb linkshare 256Kb)
queue adsl_client2_dn bandwidth 100Kb hfsc(realtime 100Kb linkshare 2048Kb)
# SDSL
# Clients get a guaranteed 1/20th of the bandwidth, and fight with their
peers
# over the remaining 19/20th.
# 2048/20=102 (rounded to whole Kb)
queue sdsl_client1_up bandwidth 100Kb hfsc(realtime 12Kb linkshare 256Kb)
queue sdsl_client1_dn bandwidth 100Kb hfsc(realtime 100Kb linkshare 2048Kb)
queue sdsl_client2_up bandwidth 100Kb hfsc(realtime 12Kb linkshare 256Kb)
queue sdsl_client2_dn bandwidth 100Kb hfsc(realtime 100Kb linkshare 2048Kb)
# Misc
# non-DSL stuff gets whatever is left over
# Default priority is 7, so make this have a lower priority
queue default_up bandwidth 2Mb priority 5 hfsc(default realtime 1Mb
linkshare 9Mb)
queue default_dn bandwidth 2Mb priority 5 hfsc(default realtime 1Mb
linkshare 9Mb)
I initially tried with the bandwidth argument set to the 2Mb/256Kb size,
thinking that this represented a maximum, but then pf complained that
the subqueues added up to more than the parent, so I tried this.
However, this way I still get:
claudas:~# pfctl -nf
/etc/pf.conf
pfctl: linkshare sc exceeds parent's sc
/etc/pf.conf:36: errors in queue definition
pfctl: linkshare sc exceeds parent's sc
/etc/pf.conf:37: errors in queue definition
pfctl: linkshare sc exceeds parent's sc
/etc/pf.conf:46: errors in queue definition
pfctl: linkshare sc exceeds parent's sc
/etc/pf.conf:51: errors in queue definition
pfctl: linkshare sc exceeds parent's sc
/etc/pf.conf:52: errors in queue definition
claudas:~# uname -a
OpenBSD claudas 4.2 GENERIC#1 amd64
claudas:~#
I know I have misunderstood how the numbers work, but I just don't know
how to make it all fit.
For those of you who have made it this far, I thank you.
Any ideas?
(Si1ent)Dave Wilson
Calomel
HFSC should work for this task. I will give it a go.
Since we are really concerned about the external interface lets just setup
that queue. The internal interface is probably 100MB or faster so and it
fast enough to not worry about right now.
First, do you have a 10MB link from the outside interface to your ISP? If
not, we need to set it to the proper number. If you tell HFSC to use more
bandwidth than you really have then you are negating HFSC in practice.
When we have the proper upload bandwidth to your ISP then we need to choose
an acceptable bandwidth just under it. 97% of the total is a rough idea of
what you are looking for. We want to make sure that our firewall is
queuing the packets and _not_ the upstream router. If we are sending
data faster than an upstream router can handle then they are queue, not
HFSC. We want to avoid upstream queuing at all costs.
These are the rules I came up with. Brief explanations follow:
#Total Upload = 10000Kb/s (queue at 97%)
altq on $ext_if bandwidth 9700kb hfsc queue { ack, us, adsl_up, sdsl_up, default_up }
queue ack bandwidth 80% priority 7 hfsc (realtime 5%)
queue us bandwidth 80% priority 6 hfsc (realtime 5%)
queue adsl_up bandwidth 256Kb priority 5 hfsc (realtime 128Kb upperlimit 256Kb) { adsl_client1_up, adsl_client2_up }
queue adsl_client1_up bandwidth 12Kb priority 2 hfsc(realtime 12Kb )
queue adsl_client2_up bandwidth 12Kb priority 1 hfsc(realtime 12Kb )
queue sdsl_up bandwidth 2048Kb priority 4 hfsc(realtime 512Kb upperlimit 2048Kb) { sdsl_client1_up, sdsl_client2_up }
queue sdsl_client1_up bandwidth 100Kb priority 2 hfsc(realtime 12Kb )
queue sdsl_client2_up bandwidth 100Kb priority 1 hfsc(realtime 12Kb )
queue default_up bandwidth 1% priority 3 hfsc (realtime 5% default)
The "ack" rule is for ACKnowlage packets that need to have the highest
priority. This is also explained at http://calomel.org/pf_hfsc.html
The "us" queue is for your company. since you pay for the connection you get
higher priority and more bandwidth including the ability to share all unused
bandwidth.
The "adsl_up" is for your adsl clients. They are guarantees 128Kb, but no
more than 256Kb. Each adsl client is guaranteed 12Kb and they each share
bandwidth up to 256Kb total.
The "sdsl_up" queue is for sdsl clients. They are guarantees 512Kb, but no
more than 2048Kb. Each sdsl client is guaranteed 12Kb and they each share
bandwidth up to 2048Kb total.
The "default_up" is for any rule that does not fit into the queues above.
They will get lowest priority and at least 5% of the total bandwidth.
To apply the queues to a rule just add "queue (us, ack)" for your company's
pass rules or "queue (adsl_client1_up)" for your first adsl client. Notice
I did not put "ack" in the adsl client. We want to limit adsl clients to
256Kb and not give them more bandwidth from the "ack" queue.
Hope this helps. I have not tested these rules, but they should work. If
you have more questions or if it works then we would be interested in
hearing from you.
Hierarchical Fair Service Curve (HFSC) of OpenBSD
http://calomel.org/pf_hfsc.html
--
Calomel @ http://calomel.org
Open Source Research and Reference
Calomel
Just a quick correction on the "bandwidth 80%" lines. Change all bandwidth
values to percentages or hard values to make it easier to work with. The
bandwidth directives can not exceed 100% of the "altq on $ext_if bandwidth
9700kb".
--
Calomel @ http://calomel.org
Open Source Research and Reference
Richard Wilson
thank you very much for your useful reply, I apologise for not replying
sooner but the nature of my job is such that when the support phone
rings, other things get put on the back burner, and it seems like
someone has put stupid in the water recently.
I have tried to run your sample rules through the parser (with the
addition of a definition of ext_if) and it didn't like it much:
si1entdave@claudas:~$ pfctl -nf ./pf.conf
pfctl: the sum of the child bandwidth higher than parent "root_vlan13"
pfctl: linkshare sc exceeds parent's sc
./pf.conf:6: errors in queue definition
pfctl: the sum of the child bandwidth higher than parent "root_vlan13"
pfctl: linkshare sc exceeds parent's sc
./pf.conf:10: errors in queue definition
parent sdsl_up not found for sdsl_client1_up
./pf.conf:11: errors in queue definition
parent sdsl_up not found for sdsl_client2_up
./pf.conf:12: errors in queue definition
I have a few questions, which I will attempt to work through below:
Calomel wrote:
> Richard,
>
> Just a quick correction on the "bandwidth 80%" lines. Change all bandwidth
> values to percentages or hard values to make it easier to work with.
If I use percentages, is that x% of the parent queue, or of the entire altq?
> The
> bandwidth directives can not exceed 100% of the "altq on $ext_if bandwidth
> 9700kb".
>
This doesn't seem to fit with what you did below...
> --
> Calomel @ http://calomel.org
> Open Source Research and Reference
>
>
> On Fri, Feb 29, 2008 at 12:24:41PM -0500, Calomel wrote:
>
>> Richard,
>>
>> HFSC should work for this task. I will give it a go.
>>
>> Since we are really concerned about the external interface lets just setup
>> that queue. The internal interface is probably 100MB or faster so and it
>> fast enough to not worry about right now.
>>
This is true, although I will have to limit the internal interface as
well, to be able to control downloads, as I believe one cannot queue
inbound traffic, so instead I must throttle outbound of the internal
interface.
>> First, do you have a 10MB link from the outside interface to your ISP? If
>> not, we need to set it to the proper number. If you tell HFSC to use more
>> bandwidth than you really have then you are negating HFSC in practice.
>>
Yes, I have 10Mb in both directions to the ISP.
>> When we have the proper upload bandwidth to your ISP then we need to choose
>> an acceptable bandwidth just under it. 97% of the total is a rough idea of
>> what you are looking for. We want to make sure that our firewall is
>> queuing the packets and _not_ the upstream router. If we are sending
>> data faster than an upstream router can handle then they are queue, not
>> HFSC. We want to avoid upstream queuing at all costs.
>>
>> These are the rules I came up with. Brief explanations follow:
>>
>> #Total Upload = 10000Kb/s (queue at 97%)
>> altq on $ext_if bandwidth 9700kb hfsc queue { ack, us, adsl_up, sdsl_up, default_up }
>> queue ack bandwidth 80% priority 7 hfsc (realtime 5%)
>> queue us bandwidth 80% priority 6 hfsc (realtime 5%)
>>
You said earlier that the bandwidth numbers must not be more than 100%
of the altq, but here we already have 160% after only two rules... I'm
confused...
>> queue adsl_up bandwidth 256Kb priority 5 hfsc (realtime 128Kb upperlimit 256Kb) { adsl_client1_up, adsl_client2_up }
>> queue adsl_client1_up bandwidth 12Kb priority 2 hfsc(realtime 12Kb )
>> queue adsl_client2_up bandwidth 12Kb priority 1 hfsc(realtime 12Kb )
>>
How does the system know from these lines that the clients are allowed
to take more than 12Kb, if it is available? I would expect some sort of
a linkshare or borrow here, looking at it as it is I would think there
was a hard max of 12Kb per client. Is the inferrence that without a
linkshare parameter or an upperlimit, the children of adsl_up will
automatically make use of the bandwidth up to the upperlimit 256Kb of
adsl_up?
>> queue sdsl_up bandwidth 2048Kb priority 4 hfsc(realtime 512Kb upperlimit 2048Kb) { sdsl_client1_up, sdsl_client2_up }
>> queue sdsl_client1_up bandwidth 100Kb priority 2 hfsc(realtime 12Kb )
>> queue sdsl_client2_up bandwidth 100Kb priority 1 hfsc(realtime 12Kb )
>>
Same question as above, where is the instruction to share the spare
bandwidth in sdsl_up?
Also, is there a particular reason why client1 and client2 have
different priorities? I'd like all clients in a pool to be equal, and
there aren't enough priorities for on each in a pool of 20 :-)
>> queue default_up bandwidth 1% priority 3 hfsc (realtime 5% default)
>>
Sorry to just reply with more questions, but I really want to understand
how its actually working rather than just copy and past stuff and plug
in numbers...
Si1entDave