help blocking malicious IPs from accessing my server

[reflection_probe]

hello, I'm evaluating web servers for a project I'm working on, and openresty looks like a good candidate.

my first question about using it would be how to properly cache large lists of IP addresses to be blocked from accessing the server. I'm doing this for abuse/bot prevention, in addition to a strict robots.txt and user-agent sniffing.

for context, I'm experimenting with access_by_lua_file for preventing access by setting ngx.status and then redirecting straight to an @error named location, which itself uses content_by_lua_file (but that's not really important).

I want to use this list for blocking IPs: https://github.com/stamparm/ipsum

obviously, that would involve a couple moving parts:

1. another service running in the background (probably not openresty) would periodically (every 24h) fetch the newest IPsum list. this could either go into a file, or directly to the server - not sure. I don't need help with cron jobs - only gettting the server to accept the new list. would prefer not to have to send a HUP to openresty and reload the entire config.
   
2. in access_by_lua*, the list would be checked somehow. maybe cache the file in memory, or in an ngx.shared dict. the most performant solution would probably be to get each worker to have its own cache to update every 24h, but I'm not sure how to do that.
   

I need help mainly with the second one. I'm not sure how to cache such a large thing, update it properly or anything like that. sure I could probably get something to work, but there's probably a good way of doing it that I wouldn't be able to figure out on my own.

note that I'm not interested in using any other services or proxies (fail2ban, CloudFlare, etc.), as I want blocked IPs to hit the @error named location.

please let me know if you have any ideas or need any additional information. thanks!

thanks,
reflection_probe

[Rainer Canavan]

On Tue, Apr 12, 2022 at 11:24 AM reflection_probe <p...@aaathats3as.com> wrote:
>
> hello, I'm evaluating web servers for a project I'm working on, and openresty looks like a good candidate.
>
> my first question about using it would be how to properly cache large lists of IP addresses to be blocked from accessing the server. I'm doing this for abuse/bot prevention, in addition to a strict robots.txt and user-agent sniffing.
>
> for context, I'm experimenting with access_by_lua_file for preventing access by setting ngx.status and then redirecting straight to an @error named location, which itself uses content_by_lua_file (but that's not really important).

As long as you don't send a 30x redirect to the client, that's fine.
You can just as well set ngx.status = 40x, set the content-type header
and simply ngx.print() your error page.

> I want to use this list for blocking IPs: https://github.com/stamparm/ipsum
>
> obviously, that would involve a couple moving parts:
>
> another service running in the background (probably not openresty) would periodically (every 24h) fetch the newest IPsum list. this could either go into a file, or directly to the server - not sure. I don't need help with cron jobs - only gettting the server to accept the new list. would prefer not to have to send a HUP to openresty and reload the entire config.
> in access_by_lua*, the list would be checked somehow. maybe cache the file in memory, or in an ngx.shared dict. the most performant solution would probably be to get each worker to have its own cache to update every 24h, but I'm not sure how to do that.
>
> I need help mainly with the second one. I'm not sure how to cache such a large thing, update it properly or anything like that. sure I could probably get something to work, but there's probably a good way of doing it that I wouldn't be able to figure out on my own.

I'd recommend an ngx.shared.dict and a second virtual host with a bit
of Lua that accepts updates to your block list e.g. via POST. That
way, you can add "evil" IPs during the day while you are under attack,
or remove IPs that you need to unblock. We only have hundreds or
thousands of IPs in our dicts, so I can't tell how millions of entries
affect performance. We also have a few subnets on our block list - we
store those in a lua-tablepool and let lua-resty-ipmatcher handle the
matching.

Rainer

[reflection_probe]

On 4/12/2022 10:24 AM, 'Rainer Canavan' via openresty-en wrote:
> As long as you don't send a 30x redirect to the client, that's fine.
> You can just as well set ngx.status = 40x, set the content-type header
> and simply ngx.print() your error page.

as of now, I'm setting ngx.status, and then running return
ngx.exec('@error'). that works flawlessly for running my error page.

> I'd recommend an ngx.shared.dict and a second virtual host with a bit
> of Lua that accepts updates to your block list e.g. via POST. That
> way, you can add "evil" IPs during the day while you are under attack,
> or remove IPs that you need to unblock. We only have hundreds or
> thousands of IPs in our dicts, so I can't tell how millions of entries
> affect performance. We also have a few subnets on our block list - we
> store those in a lua-tablepool and let lua-resty-ipmatcher handle the
> matching.

alright. so every 24h or so I'd fetch the newest IPsum list and then
POST it to some endpoint (behind authorization of course) that puts it
in ngx.shared? what's the performance of those dictionaries when
many-megabyte strings are being used? my main concern is that getting
those strings into the Lua VM will be a bottleneck, which is why I
wanted to cache it locally for each worker process. perhaps I could use
a global variable (explicitly with _G) - OpenResty discourages that, but
it's probably fine. will avoid doing that, though, until I can come up
with a solid method of knowing when to re-fetch it from the dictionary.

reflection_probe

[Joseph Martinez]

Hello, when are you going to pay your debts? 

--
You received this message because you are subscribed to the Google Groups "openresty-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openresty-en...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openresty-en/2335e72b-18a0-01fa-589c-4f2ea1d5814f%40aaathats3as.com.

[reflection_probe]

On 4/12/2022 6:04 PM, Joseph Martinez wrote:
> Hello, when are you going to pay your debts?

go away, spammer...

[reflection_probe]

nginx has supported http/3 for a while, but it's disabled by default and
has to be enabled with the --with-http_v3_module configure flag. it
requires a couple dependencies as well (quiche and BoringSSL), but
that's really it.

over 25% of the top 10 million websites support HTTP/3 (according to
Wikipedia), and I feel like openresty is unnecessarily limiting my
options. since HTTP/3 was not enabled at build time, I can't use it at
all without recompiling nginx with all of openresty's additional patches
and modules, which is going to be extremely difficult.

it would be much easier for openresty to add the flag to their builds so
that everyone can opt-in to HTTP/3 support if they want it. it's not
used by default with the default configuration, so you won't be changing
existing setups, just unlocking new functionality.

I'd strongly urge you to consider adding HTTP/3 support to your next
release.

thanks,
reflection_probe

[reflection_probe]

P.S. I am terribly sorry for the duplicate post, and am currently
cursing my email client and wishing that I had the ability to delete
that other post that it made. that should not have been a reply. if any
group administrator sees this please delete my prior duplicate as that
was a mistake.

[reflection_probe]

this is dumb. I can't even post a new thread correctly anymore to
correct my mistake. screw mailing lists.