PF inadequacy: queue download

74 views
Skip to first unread message

kesta...@gmail.com

unread,
Apr 29, 2006, 9:04:51 AM4/29/06
to
Why can't you queue download traffic on an interface? The reason
openbsd.org's FAQ gives is:

"Note that queueing is only useful for packets in the outbound
direction. Once a packet arrives on an interface in the inbound
direction it's already too late to queue it -- it's already consumed
network bandwidth to get to the interface that just received it. The
only solution is to enable queueing on the adjacent router or, if the
host that received the packet is acting as a router, to enable queueing
on the internal interface where packets exit the router."

But this is wrong. It's not too late to queue it; by queueing it and
dropping some packets of inbound traffic the sending host slows down
the speed at which it sends.

I'm using pf to do NAT on my box, and I can shape download traffic
using the 'queueing on the internal interface' hack; so why can't I do
it elegantly on one interface?
Shaping NAT traffic downloads works fine with this hack, but I also run
some services on the external interface. With downloads queued on the
internal interface there's no way to queue the services' download
traffic, which means an external service can hog up all the bandwidth
and I can't do anything.


I know this is possible because IPFW with dummynet doesn't have any
problems. If everyone loves PF because of its elegance why can't it do
something as simple as queue download traffic?

Regards,
Kestas

Steven Schneider

unread,
Apr 29, 2006, 12:32:17 PM4/29/06
to
* kesta...@gmail.com <kesta...@gmail.com> [2006-04-29]:

> Why can't you queue download traffic on an interface? The reason
> openbsd.org's FAQ gives is:
>
> "Note that queueing is only useful for packets in the outbound
> direction. Once a packet arrives on an interface in the inbound
> direction it's already too late to queue it -- it's already consumed
> network bandwidth to get to the interface that just received it. The
> only solution is to enable queueing on the adjacent router or, if the
> host that received the packet is acting as a router, to enable queueing
> on the internal interface where packets exit the router."
>
> But this is wrong. It's not too late to queue it; by queueing it and
> dropping some packets of inbound traffic the sending host slows down
> the speed at which it sends.
>
Are you sure about that? Perhaps a well behaved sending host
would, but what if it isn't? Also, if you're being DDOSd, will
this even matter?

> Shaping NAT traffic downloads works fine with this hack, but I also run
> some services on the external interface. With downloads queued on the
> internal interface there's no way to queue the services' download
> traffic, which means an external service can hog up all the bandwidth
> and I can't do anything.
>

I haven't heard of any firewall that successfully could. If you're
being DDOSd, you're being DDOSd. No firewall with any special set
of rules can improve your bandwidth in that case. If the pipe is
filled, it's filled.


>
> I know this is possible because IPFW with dummynet doesn't have any
> problems.

I don't use IPFW, so I can't speak on its specific technical merits.
Maybe you should ask the IPFW devs how they're able to perform this
magic. Last I heard, the PF devs were technicians, engineers, and
scientists, not a single magic-user among them. (IRL anyhow. :-))

If everyone loves PF because of its elegance why can't it do
> something as simple as queue download traffic?
>

I think that you _can_ configure PF to do this, but I believe that
what the developers are trying to say is, `What's the point'? We're
talking about trying to control traffic _before_ it hits your
interface. Even if the remote sending host is well-behaved enough
to slow down its sending rate, it still has to interact with PF
before PF can decide whether to pass the packets, drop the packets,
or tell the sending host to `bugger off'.

So, you can queue the download traffic, but that really has a minor
to no effect on traffic outside of your firewall. The queuing
actually occurs on _your_ side of the external interface.

My $0.02 CDN. Take it for what it's worth, or exchange it for
better currency. :-)

--
W. Steven Schneider <steven_s...@telus.net>

kesta...@gmail.com

unread,
Apr 29, 2006, 7:26:30 PM4/29/06
to
> I haven't heard of any firewall that successfully could. If you're
> being DDOSd, you're being DDOSd. No firewall with any special set
> of rules can improve your bandwidth in that case. If the pipe is
> filled, it's filled.
Yes, if you're being DDoSed then incoming traffic shaping won't do
anything, but if you're using TCP streams from cooperative hosts you
can shape incoming traffic very effectively; you drop packets, sender
realises packets are getting lost, sender slows down sending packets.
It works when you use the hack of queueing on the internal interface
when you're using NAT, it clearly works, so why can't you do it on a
single interface?

> I don't use IPFW, so I can't speak on its specific technical merits.
> Maybe you should ask the IPFW devs how they're able to perform this
> magic. Last I heard, the PF devs were technicians, engineers, and
> scientists, not a single magic-user among them. (IRL anyhow. :-))

There's no magic about it :-P

> I think that you _can_ configure PF to do this, but I believe that
> what the developers are trying to say is, `What's the point'? We're
> talking about trying to control traffic _before_ it hits your
> interface. Even if the remote sending host is well-behaved enough
> to slow down its sending rate, it still has to interact with PF
> before PF can decide whether to pass the packets, drop the packets,
> or tell the sending host to `bugger off'.

> So, you can queue the download traffic, but that really has a minor
> to no effect on traffic outside of your firewall. The queuing
> actually occurs on _your_ side of the external interface.

As I explained above shaping download traffic works great, but only
using the two interface hack. People on other boards are telling me I
should put another box beetween the LAN and the internet so I can use
the two interface hack for all traffic, but this seems stupid; if it
works over two interfaces so it can certainly work on one interface, so
why doesn't it?

Kesas

Steven Schneider

unread,
Apr 29, 2006, 8:54:48 PM4/29/06
to
* kesta...@gmail.com <kesta...@gmail.com> [2006-04-29]:

>> I haven't heard of any firewall that successfully could. If you're
>> being DDOSd, you're being DDOSd. No firewall with any special set
>> of rules can improve your bandwidth in that case. If the pipe is
>> filled, it's filled.
> Yes, if you're being DDoSed then incoming traffic shaping won't do
> anything, but if you're using TCP streams from cooperative hosts you
> can shape incoming traffic very effectively; you drop packets, sender
> realises packets are getting lost, sender slows down sending packets.
> It works when you use the hack of queueing on the internal interface
> when you're using NAT, it clearly works, so why can't you do it on a
> single interface?
>
Obviously, I misunderstood. It's still true that the queuing
actually occurs on your side of the firewall, and that it has to be
on the outbound traffic.

Now I might still be wrong, but I think that you want to control
your machine's ack rate. Have you looked at
http://www.benzedrine.cx/ackpri.html? This guy has some ideas that
might be applicable to your situation.

You may also want to look at
http://www.bgnett.no/~peter/pf/en/long-firewall.html.

I've found both of these sites to be sources of useful PF info. :-)

kesta...@gmail.com

unread,
Apr 30, 2006, 2:39:24 AM4/30/06
to
Thanks for the help, but I still don't think you understand. This isn't
a limitation in my understanding of PF and ALTQ, this is a limitation
of PF and ALTQ itself; you cannot shape traffic coming in on an
interface. There's no good reason why you shouldn't be able to do this.
I want to know how I can get this added to a todo list, everyone who
has a firm grasp of PF and ALTQ, and isn't a zealot, agrees that this
is an important feature which PF lacks, and should be implemented asap.

Kestas

Marco S Hyman

unread,
Apr 30, 2006, 5:39:02 PM4/30/06
to
kesta...@gmail.com writes:

> Thanks for the help, but I still don't think you understand. This isn't
> a limitation in my understanding of PF and ALTQ, this is a limitation
> of PF and ALTQ itself; you cannot shape traffic coming in on an
> interface. There's no good reason why you shouldn't be able to do this.

No, there is plenty of good reason to not do so. Spending extra
processor cycles queuing (i.e. delaying) packets that have already
been received, e.g. have already used network bandwidth, is not a
good thing. What you are asking is "please make my network slower
so well-behaved peers will throttle their long lived TCP connections"
The cost of doing that will slow *everything else* down, too. It buys
you nothing with UDP, ICMP, and mis-behaved TCP hosts (of which there
are all too many). It exacerbates DOS scenarios. It makes things
like popular web servers slower -- http requests are too short
lived to get much benefit from TCP congestion control.

> I want to know how I can get this added to a todo list, everyone who
> has a firm grasp of PF and ALTQ, and isn't a zealot, agrees that this
> is an important feature which PF lacks, and should be implemented asap.

Since you appear to be the only person who is eager to slow down your
network processing for minimal benefit (if any at all) you will probably
have to do the work yourself.

// marc

kesta...@gmail.com

unread,
Apr 30, 2006, 7:33:45 PM4/30/06
to
Most hosts are well behaved, and UDP also throttles if the protocol is
intelligent. The fact that this is useful is obvious because so many
people do it on their gateway using two interfaces. I know it's useful
because before implementing it one person downloading would hog up the
whole network; with incoming traffic shaping the person hogging up the
network immidiately gets throttled down and other queues can get a
chance.

Extra processor cycles? Downloading at full pace my Pentium II 350MHz
isn't using 5% of it processor.

Finally I'm not the only person who wants this. From the thread in
bit.listserv.openbsd-pf:
> It's a question of what's "useful". Not useful against
> a malicious agent, but useful for traffic shaping regardless.

> It surely has its caveats, but has its use, too. I'm stuck with ipfw on
> one last machine, because I can't limit bulk TCP traffic with pf.

> I can speak for myself - I can't afford both the hardware and the
> electricity bill for a separate machine. Maybe downstream limiting isn't
> very robust, but IMO is the biggest thing pf/altq lacks.

I think most people who use pf for anything more than the most trivial
traffic shaping applications realise that this is a major feature which
is lacking.

Kestas

Steven Schneider

unread,
Apr 30, 2006, 11:43:43 PM4/30/06
to
* kesta...@gmail.com <kesta...@gmail.com> [2006-04-30]:

> Thanks for the help, but I still don't think you understand.

I spent a fair bit of today researching the issue to make sure that
I understand you better. I believe I do now.

> This isn't
> a limitation in my understanding of PF and ALTQ, this is a limitation
> of PF and ALTQ itself; you cannot shape traffic coming in on an
> interface.

Yes, I was aware that it can't and that this is a limitation of
PF/ALTQ. I'm not entirely convinced that it's serious one, however.

Do you have access to any documents that show measurable advantages
that queuing incoming traffic would have over not queuing it?
Personally, I've been finding mostly "ad hoc" accounts being cited
as evidence for shaping incoming traffic.

> There's no good reason why you shouldn't be able to do this.
> I want to know how I can get this added to a todo list, everyone who
> has a firm grasp of PF and ALTQ, and isn't a zealot, agrees that this
> is an important feature which PF lacks, and should be implemented asap.
>

Your statement assumes that anyone who disagrees with you has a poor
grasp of PF/ALTQ, and is a zealot. That polarizes things just a
bit, don't you think?

Keeping in mind that I'm not a developer myself, the following
points might be helpful.

If you could demonstrate the importance of shaping incoming traffic
to Theo and the developers. Better yet, submit a patch, that might
go further than simply telling them that it, "should be implemented
asap."

Generous donations may also help. I hear that they're rather fond
of a good beer. :-)

kesta...@gmail.com

unread,
May 1, 2006, 7:35:36 AM5/1/06
to
> I spent a fair bit of today researching the issue to make sure that
> I understand you better. I believe I do now.
Thanks, I appreciate that you'd spend some time trying to understand
this.

> Do you have access to any documents that show measurable advantages
> that queuing incoming traffic would have over not queuing it?
> Personally, I've been finding mostly "ad hoc" accounts being cited
> as evidence for shaping incoming traffic.

Hmm, okay.
These images show me downloading two files on two different hosts. I
want one host to recieve 30KB/s, and the other to recieve the rest. But
most importantly I want it to happen immidiately (so the priority queue
can be used to quickly get web pages). I run a download on the low
priority host, and start a download on the high priority host, and run
date before and after, and see the time it takes to stabilise at a fair
speed for each.
Here it is without download queueing:
http://img151.imagevenue.com/img.php?loc=loc30&image=82445_asdf2.JPG
It takes over 20 seconds for the speeds to stabilise at around 25KB/s
each. This means if someone's downloading it'll take over 20 seconds to
download a fairly lightweight webpage.

And with download queueing:
http://img41.imagevenue.com/img.php?loc=loc280&image=82210_asdf.JPG
It takes 2 seconds.

Remember people do use download queueing using the two interface hack I
use, but this functionality is limited only to NAT or dedicated traffic
shaping boxes; I've shown you why it's useful, and lots of people use
it with NAT, so why can't you do it on one interface?

> Your statement assumes that anyone who disagrees with you has a poor
> grasp of PF/ALTQ, and is a zealot. That polarizes things just a
> bit, don't you think?

I don't know how a statement can assume anything, or what is polarized,
but yes that's true. It's not a personal attack to say that I doubt
you're familiar with PF's traffic shaping capabilities, I'm not
familiar with lots of things. I think anyone who understands PF's
traffic shaping capabilities must understand its limitations, and this
is a severe limitation.

> If you could demonstrate the importance of shaping incoming traffic
> to Theo and the developers. Better yet, submit a patch, that might
> go further than simply telling them that it, "should be implemented
> asap."

I've done the former, not that it needs to be done as I'm sure the
developers understand this download shaping limitation intimately, and
unfortunately I can't do the latter as I don't know C very well, and I
think working on something as complex and mission critical as PF+ALTQ
would be daunting to someone who knew C well.

> Generous donations may also help. I hear that they're rather fond
> of a good beer. :-)

I said I'd be happy to donate in another thread, to help get this moved
along faster. Maybe a couple of days wages, but the only person who
responded suggested a fee of $20,000. I'm a student earning minimum
wage, I'm the last person who could splash out /$20,000/ to fund a
hackathon to get this feature re-implemented.

Kestas

Reply all
Reply to author
Forward
0 new messages