synchronet behind a reverse proxy

[martylake]

Dear all,

This is my first message ever written on ssh to any bulletin board \o/ So excuse my lack of etiquette.

I am trying to host different services on a rpi4, at my place behind the ISP's router. In order to differentiate between different website, I use a router called Traefik and redirects the traffic to synchronet. It all works great when I am alone, and I am planning to try to invite friends.

Then I realised, what if someone pounds at sbbs too hard to hack it ? Fortunately there is some retry mecanism and ban too. The issue is that in the logs, I don't see the final IP, but instead the local IP of Traefik.

Do you have an idea of how to configure this ? With some other service that uses php for example, I had to install mod_remoteip and tune the LogFormat in order to display the original IP.

Best,
martylake
--- Synchronet 3.19a-Win32 NewsLink 1.113
* Vertrauen - Riverside County, California - telnet://vert.synchro.net

[martylake]

To: martylake

> Do you have an idea of how to configure this ?

To be more explicit: how to configure synchronet such as the original IP is
considered for bans, instead of the local ones ?

[Digital Man]

To: martylake
Re: Re: synchronet behind a reverse proxy
By: martylake to martylake on Sun Nov 21 2021 11:15 am

>
> > Do you have an idea of how to configure this ?
>
> To be more explicit: how to configure synchronet such as the original IP is
> considered for bans, instead of the local ones ?

If Synchronet is sitting behind a proxy, you'd have to do so magic with the proxy to provide the actual remote/client IP address to Synchronet. Kind of like this:
https://wiki.synchro.net/howto:haproxy
--
digital man

Synchronet/BBS Terminology Definition #70:
SIGHUP = Hangup signal sent to a process when its controlling terminal is closed
Norco, CA WX: 77.6øF, 16.0% humidity, 12 mph N wind, 0.00 inches rain/24hrs

[martylake]

To: Digital Man

> If Synchronet is sitting behind a proxy, you'd have to do so magic with the
> proxy to provide the actual remote/client IP address to Synchronet. Kind of
> like this:
> https://wiki.synchro.net/howto:haproxy

Thank you very much for your swift answer, I wonder how I did not find this on
the wiki before, maybe I was using the wrong keywords.

Self-hosting is such a journey ! Everytime something is broken, I realize it's
my understanding of the whole setup that's broken too and I have to learn
something new.

[martylake]

To: martylake

> Self-hosting is such a journey ! Everytime something is broken, I realize
> it's my understanding of the whole setup that's broken too and I have to
> learn something new.

I am in the process of replacing traefik with haproxy. I got ssh, webv4 and
telnet working so far.

ws is a bit problematic because haproxy (from 2.1
https://www.haproxy.com/fr/blog/haproxy-2-1/ ) Defaulted HTTP Representation to
HTX . Which means it feeds lowercased headers, and one needs to override this
behavior by providing a dictionnary of headers not to be lowercased, and this
insulting option h1-case-adjust-bogus-server

With that solved, if I disable synchronet's HA_PROXY, the ws telnet works
great. But if I enable HA_PROXY, term is complaining that ws did not provide
enough information and shut downs the connexion.

Here are some logs (is it good etiquette to paste logs like that in a post?),
do you have an idea of what piece of configuration might be missing there ?

```
11/24 09:16:12 srvc 0037 WS connection accepted from: 172.18.0.8 port 44168
11/24 09:16:12 srvc 0037 WS JavaScript service thread started
11/24 09:16:12 srvc 0037 WS Handshake Line: GET / HTTP/1.1
11/24 09:16:12 srvc 0037 WS Handshake Line: Host: forum.talbot.audio:11235
11/24 09:16:12 srvc 0037 WS Handshake Line: user-agent: Mozilla/5.0 (X11;
Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0
11/24 09:16:12 srvc 0037 WS Handshake Line: accept: */*
11/24 09:16:12 srvc 0037 WS Handshake Line: accept-language:
fr,en-US;q=0.7,en;q=0.3
11/24 09:16:12 srvc 0037 WS Handshake Line: accept-encoding: gzip, deflate, br
11/24 09:16:12 srvc 0037 WS Handshake Line: Sec-WebSocket-Version: 13
11/24 09:16:12 srvc 0037 WS Handshake Line: Origin: https://forum.talbot.audio
11/24 09:16:12 srvc 0037 WS Handshake Line: Sec-WebSocket-Protocol: binary,
base64, plain
11/24 09:16:12 srvc 0037 WS Handshake Line: sec-websocket-extensions:
permessage-deflate
11/24 09:16:12 srvc 0037 WS Handshake Line: Sec-WebSocket-Key:
U0lWMoq6Dw1Cr+ymvKzytw==
11/24 09:16:12 srvc 0037 WS Handshake Line: dnt: 1
11/24 09:16:12 srvc 0037 WS Handshake Line: Connection: Upgrade
11/24 09:16:12 srvc 0037 WS Handshake Line: sec-fetch-dest: websocket
11/24 09:16:12 srvc 0037 WS Handshake Line: sec-fetch-mode: websocket
11/24 09:16:12 srvc 0037 WS Handshake Line: sec-fetch-site: same-site
11/24 09:16:12 srvc 0037 WS Handshake Line: pragma: no-cache
11/24 09:16:12 srvc 0037 WS Handshake Line: cache-control: no-cache
11/24 09:16:12 srvc 0037 WS Handshake Line: Upgrade: websocket
11/24 09:16:12 srvc 0037 WS Handshake Line: X-Forwarded-For: 192.168.1.254
11/24 09:16:12 srvc 0037 WS Handshake Line:
11/24 09:16:12 srvc 0037 WS Connecting to localhost:23
11/24 09:16:12 term 0040 Working out client address from HAProxy PROTO
11/24 09:16:13 term 0040 multisock read_socket() - No data?
11/24 09:16:13 term 0040 * HAPROXY looking for version - failed
[fffc01fffb17fffb1c]
11/24 09:16:13 srvc 0037 WS Server socket no longer connected
11/24 09:16:13 srvc 0037 WS service thread terminated (1 clients remain, 1
total, 3 served)
nov. 24 10:16:13 alfred a02100129bc0[609]: 192.168.1.254:32852
[24/Nov/2021:09:16:12.448] fe_http~ be_ws_sbbs/sbbs_ws 0/0/2/43/1131 101 187 -
- ---- 3/3/0/0/0 0/0 "GET / HTTP/1.1"
```

[deon]

To: martylake
Re: Re: synchronet behind a reverse proxy

By: martylake to martylake on Wed Nov 24 2021 01:23 am

Howdy,

> With that solved, if I disable synchronet's HA_PROXY, the ws telnet works great. But if I enable HA_PROXY, term is complaining that ws
> did not provide enough information and shut downs the connexion.

> 11/24 09:16:13 term 0040 multisock read_socket() - No data?
> 11/24 09:16:13 term 0040 * HAPROXY looking for version - failed [fffc01fffb17fffb1c]

Is your syncrhonet running on Windows by chance?


...лоеп

---
ю Synchronet ю Alterant | an SBBS in Docker on Pi!

[martylake]

To: deon

Re: Re: synchronet behind a reverse proxy

By: deon to martylake on Wed Nov 24 2021 08:51 pm

> Howdy,
\o/

> Is your syncrhonet running on Windows by chance?

Unfortunately not. It's running on raspberrypi4 raspbian under docker.

[deon]

To: martylake

Re: Re: synchronet behind a reverse proxy

By: martylake to martylake on Wed Nov 24 2021 01:23 am

Hey Marty,

> I am in the process of replacing traefik with haproxy. I got ssh, webv4 and telnet working so far.
>
> ws is a bit problematic because haproxy (from 2.1 https://www.haproxy.com/fr/blog/haproxy-2-1/ ) Defaulted HTTP Representation to HTX . Which means it feeds lowercased headers, and one needs to override this behavior by providing a dictionnary of
> headers not to be lowercased, and this insulting option h1-case-adjust-bogus-server
>
> With that solved, if I disable synchronet's HA_PROXY, the ws telnet works great. But if I enable HA_PROXY, term is complaining that ws did not provide enough information and shut downs the connexion.

I dont use websockets, so I dont know the extent to which it can be configured.

Your problem is this

> 11/24 09:16:12 srvc 0037 WS Connecting to localhost:23

When you enable HAPROXY, the only connections that can come in from port 23 are *from* haproxy. If the websockets is connecting directly to port 23, then the haproxy code will drop the connection, as seen by:

> 11/24 09:16:12 term 0040 Working out client address from HAProxy PROTO
> 11/24 09:16:13 term 0040 multisock read_socket() - No data?

So reconfigure the websockets part to connect to haproxy's port, not SBBS directly.

...лоеп

---
ю Synchronet ю Alterant | an SBBS in Docker on Pi!

[deon]

To: Digital Man

Re: Re: synchronet behind a reverse proxy
By: martylake to martylake on Wed Nov 24 2021 01:23 am

Hey DM,

> 11/24 09:16:13 term 0040 * HAPROXY looking for version - failed [fffc01fffb17fffb1c]

These are Telnet IAC codes, which I guess is "often" received from the remote after a first connect? (Well for telnet anyway).

Might be worthwhile adding something around https://gitlab.synchro.net/main/sbbs/-/blob/master/src/xpdev/multisock.c#L420, where if we receive ff, and a valid IAC command (https://users.cs.cf.ac.uk/Dave.Marshall/Internet/node141.html) (ie: greater than 240) and a 3rd char? (as the first 3 words received), that the error message display

"Looks like you are connecting directly to the [telnet] port - you cant do that with HAProxy enabled."

There might be something similarly appropriate for rlogin and web?

Not sure if that will help (the end user at least) diagnose some errors themselves...

...лоеп

---
ю Synchronet ю Alterant | an SBBS in Docker on Pi!

[martylake]

To: deon

Re: Re: synchronet behind a reverse proxy

By: deon to martylake on Thu Nov 25 2021 10:43 am

> I dont use websockets, so I dont know the extent to which it can be
> configured.

I have tried to look at the source code, but did not find the way to configure this.

> So reconfigure the websockets part to connect to haproxy's port, not SBBS
> directly.

The issue is that websocket is a protocol on top of http, that's why we need a websocket->telnet proxy. The issue if I understand correctly is that this proxy is not forwarding haproxy information to the backend.

Or maybe there is a way to configure haproxy directly to act as a websocket proxy for us ?

[echicken]

To: martylake

Re: Re: synchronet behind a reverse proxy

By: martylake to deon on Thu Nov 25 2021 10:23:07

ma> The issue is that websocket is a protocol on top of http, that's why we
ma> need a websocket->telnet proxy. The issue if I understand correctly is
ma> that this proxy is not forwarding haproxy information to the backend.

The websocket service is definitely not using the HAProxy protocol in its connection to the telnet server, which is what would ultimately need to happen, somehow.

I gather that HAProxy has a couple of different modes - one for generic TCP services, using its own protocol, and another as a conventional HTTP reverse proxy. Please correct me if I'm wrong on that.

Assuming it's acting as a typical HTTP reverse proxy where it interacts with the Synchronet web and websocket services, it should be sending the 'x-forwarded-for' header along to the upstream. If so, we would need to make the websocket server:

1) Capture the value of the x-forwarded-for header
2) Check if the telnet server has the HAPROXY_PROTO flag in sbbs.ini
3) Speak the HAProxy protocol when connecting to the telnet server

Parts 1 & 2 are simple. A glance at HAProxy's docs suggest that 3 is no big deal either, but I may be missing something.

If it sounds like I'm on the right track, I can make a branch that we can test this in. I'd rather avoid setting up HAProxy here, but I can make some changes and you can try them.

ma> Or maybe there is a way to configure haproxy directly to act as a
ma> websocket proxy for us ?

Not that I can see, unfortunately - but that would be simplest if possible.

---
echicken
electronic chicken bbs - bbs.electronicchicken.com
---
ş Synchronet ş electronic chicken bbs - bbs.electronicchicken.com

[martylake]

To: echicken

Re: Re: synchronet behind a reverse proxy

By: echicken to martylake on Thu Nov 25 2021 07:25 pm

Hello echicken,

> I gather that HAProxy has a couple of different modes - one for generic TCP
> services, using its own protocol, and another as a conventional HTTP reverse
> proxy. Please correct me if I'm wrong on that.

As far as I understand it, it's particularly useful for TCP services. I don't know if it also works for HTTP reverse proxy, the haproxy that I use at the moment don't need the `check-proxy v2` option for the magic header x-forwarded-for to happen. Instead it just needs the option `forwarder-for`.
As far as I understand too yes, I just discovered this mode two days ago.

> If it sounds like I'm on the right track, I can make a branch that we can
> test this in. I'd rather avoid setting up HAProxy here, but I can make some
> changes and you can try them.

It sounds like you are on the right track yes ! I am willing to test whatever changes you do, however I need to setup something to not depend on docker, unless you can ship the changes as docker images for armv7 too ?

I just compiled synchronet locally, but if I run it, it complains about missing configuration files. I should RTFM to figure out how to properly do it.

I guess I will be able to pull changes that you push on a branch.

[echicken]

To: martylake

Re: Re: synchronet behind a reverse proxy

By: martylake to echicken on Thu Nov 25 2021 12:21:45

ma> at the moment don't need the `check-proxy v2` option for the magic header
ma> x-forwarded-for to happen. Instead it just needs the option
ma> `forwarder-for`. As far as I understand too yes, I just discovered this
ma> mode two days ago.

If you want to check if this is really happening, you can put this file somewhere on your Synchronet webserver as 'headers.xjs', then browse to it and see what it spits out:

<!DOCTYPE html>
<html lang="en">
<head>
<title>Headers</title>
<meta charset="UTF-8">
</head>
<body>
<? Object.keys(http_request.header).forEach(function (e) { ?>
<? write(e); ?>: <? write(http_request.header[e]); ?><br>
<? }); ?>
</body>
</html>

One of the rows should show the x-forwarded-for header.

ma> I just compiled synchronet locally, but if I run it, it complains about
ma> missing configuration files. I should RTFM to figure out how to properly
ma> do it.

ma> I guess I will be able to pull changes that you push on a branch.

Sorry, I should have asked if you did a development / git-based installation in the first place. Once you have one working, let me know. I'll tinker with this a bit tonight if I'm able, or tomorrow.

---
echicken
electronic chicken bbs - bbs.electronicchicken.com
---
ş Synchronet ş electronic chicken bbs - bbs.electronicchicken.com

[martylake]

To: echicken

> If you want to check if this is really happening, you can put this file
> somewhere on your Synchronet webserver as 'headers.xjs', then browse to it
> and see what it spits out:

host: REDACTED

user-agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101
Firefox/94.0

accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/
webp,*/*;q=0.8
accept-language: fr,en-US;q=0.7,en;q=0.3
accept-encoding: gzip, deflate, br
dnt: 1
upgrade-insecure-requests: 1
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: none
sec-fetch-user: ?1
x-forwarded-for: 192.168.1.254

> One of the rows should show the x-forwarded-for header.

yes there is one !

> Sorry, I should have asked if you did a development / git-based installation
> in the first place. Once you have one working, let me know. I'll tinker with
> this a bit tonight if I'm able, or tomorrow.

Ok, I will focus on getting a git-based installation ready ASAP and let you
know. Thank you very much for offering help on that !

[deon]

To: martylake

Re: Re: synchronet behind a reverse proxy

By: martylake to deon on Thu Nov 25 2021 10:23 am

> > So reconfigure the websockets part to connect to haproxy's port, not SBBS directly.
>
> The issue is that websocket is a protocol on top of http, that's why we need a websocket->telnet proxy. The issue if I understand correctly is that this proxy is not forwarding haproxy information to the backend.

Correct, there are two paths that could address this.

* The websocket connects to haproxy, not SBBS directly.

* The websocket speaks that haproxy protocol when talking with SBBS directly.

The first is probably easy to do, if you can configure it via a config file (sorry I dont know).

The second would require development I'm guessing...

...лоеп

---
ю Synchronet ю Alterant | an SBBS in Docker on Pi!

[deon]

To: echicken

Re: Re: synchronet behind a reverse proxy

By: echicken to martylake on Thu Nov 25 2021 07:25 pm

> Assuming it's acting as a typical HTTP reverse proxy where it interacts with the Synchronet web and websocket services, it should be sending the 'x-forwarded-for' header along to the upstream. If so, we would need to make the websocket server:
>
> 1) Capture the value of the x-forwarded-for header
> 2) Check if the telnet server has the HAPROXY_PROTO flag in sbbs.ini
> 3) Speak the HAProxy protocol when connecting to the telnet server
>
> Parts 1 & 2 are simple. A glance at HAProxy's docs suggest that 3 is no big deal either, but I may be missing something.

Yup, that will do it.

3) is easy - you should be able to talk the v1 or v2 protocol, there is a link to it in the comments of the source code. Its basically the first string of chars when the connection is made, before doing actual telnet stuff.

...лоеп

---
ю Synchronet ю Alterant | an SBBS in Docker on Pi!

[echicken]

To: martylake

Re: Re: synchronet behind a reverse proxy

By: martylake to echicken on Thu Nov 25 2021 13:04:08

ma> Ok, I will focus on getting a git-based installation ready ASAP and let
ma> you know. Thank you very much for offering help on that !

My next couple of questions:

Is HAProxy reverse-proxying websocket connections to Synchronet's websocket service? This needs to be configured in addition to reverse proxying traffic to Synchronet's webserver. (Hint: you probably had to set values for 'wsp' and
'wssp' in the [web] section of 'ctrl/modopts.ini' if you set this up.)

You mentioned something in an earlier message about HAProxy lowercasing the names of HTTP headers. If the client sends 'Some-Header: Some Data', can I expect the upstream to receive 'some-header: Some Data'? This will become important, because it looks like the websocket service currently does case-sensitive matching on header lines.

---
echicken
electronic chicken bbs - bbs.electronicchicken.com
---
ş Synchronet ş electronic chicken bbs - bbs.electronicchicken.com

[martylake]

To: echicken

Re: Re: synchronet behind a reverse proxy

By: echicken to martylake on Fri Nov 26 2021 04:08 am

Dear echicken,

> Is HAProxy reverse-proxying websocket connections to Synchronet's websocket
> service? This needs to be configured in addition to reverse proxying traffic
> to Synchronet's webserver. (Hint: you probably had to set values for 'wsp'
> and 'wssp' in the [web] section of 'ctrl/modopts.ini' if you set this up.)

Yes it currently is, it looks like this:
The_Internet -> ISP's router -> haproxy -> wss port of sbbs -> telnet sbbs

If I disable sbbs' HAPROXY_PROTO, the web telnet wrapper works great.

> You mentioned something in an earlier message about HAProxy lowercasing the
> names of HTTP headers. If the client sends 'Some-Header: Some Data', can I
> expect the upstream to receive 'some-header: Some Data'? This will become
> important, because it looks like the websocket service currently does
> case-sensitive matching on header lines.

There are a couple of settings in haproxy to massage the headers so they arrive in the expected capitalization.

By default, haproxy would lowercase all headers, the way you described in your example `Some-Header: Some Data` -> haproxy -> `some-header: Some Data`. If it's not too much trouble, changing from case-sensitive matching to case-insensitive matching for headers would make the websocket service more compatible, but it is not mandatory at all, as I said, haproxy also has options to specify the required header capitalization per route.

When I get this whole setup working, I can suggest a few edits to synchronet's wiki's proxy page to account for all those peculiarities.

[martylake]

To: deon

Re: Re: synchronet behind a reverse proxy

By: deon to martylake on Fri Nov 26 2021 10:38 am

> The first is probably easy to do, if you can configure it via a config file
> (sorry I dont know).

Unfortunately, there seems to be lots of options for forwarding websocket to a websocket server:
internet-ws -> haproxy -> server ws
but no option for proxying:
internet-ws -> haproxy -> raw tcp with haproxy_proto preamble

> The second would require development I'm guessing...

Hopefully, it is less development than creating a new ws->tcp with haproxy_proto preamble support ?

[deon]

To: martylake

Re: Re: synchronet behind a reverse proxy

By: martylake to deon on Thu Nov 25 2021 08:34 pm

Howdy,

> > The first is probably easy to do, if you can configure it via a config file (sorry I dont know).
>
> Unfortunately, there seems to be lots of options for forwarding websocket to a websocket server:
> internet-ws -> haproxy -> server ws
> but no option for proxying:
> internet-ws -> haproxy -> raw tcp with haproxy_proto preamble

Not sure we are talking the same thing...

For normal telnet it is this:

telnet -> haproxy (telnet port) -> sbbs (telnet port).

So, for websocket telnet client, it would be:

browser -> haproxy (http port) -> websocket -> haproxy (telnet port) -> sbbs (telnet port)

Or, if websocket can talk haproxy proto:

browser -> haproxy (http port) -> websocket -> sbbs (telnet port with haproxy proto introduction)

Looks like echicken was working on the last item - I would have thought for the first one, you can change "localhost:23" to "haproxy:23"?

...лоеп

---
ю Synchronet ю Alterant | an SBBS in Docker on Pi!

[martylake]

To: deon

Re: Re: synchronet behind a reverse proxy

By: deon to martylake on Fri Nov 26 2021 04:22 pm

Dear deon,

> browser -> haproxy (http port) -> websocket -> haproxy (telnet port) -> sbbs
> (telnet port)

> Looks like echicken was working on the last item - I would have thought for
> the first one, you can change "localhost:23" to "haproxy:23"?

I misunderstood, indeed ! I think proposition 2 it equivalent to proposition 3 because if websocket does not forward haproxy instructions, there is no way to retrieve back this information down the chain:

browser -> haproxy (http port, adds http header x-forwarded-from) -> websocket (currently discards information) -> ...whatever comes after does not have the information anymore ?

I hope contributing as guinea pig for echicken helps. The spec of proxy_protool does not look so long, but I have never implemented such network specs before so I˙am not sure if my intuition is correct there.

[martylake]

To: echicken

Re: Re: synchronet behind a reverse proxy

By: echicken to martylake on Thu Nov 25 2021 08:53 pm

> Sorry, I should have asked if you did a development / git-based installation
> in the first place. Once you have one working, let me know. I'll tinker with
> this a bit tonight if I'm able, or tomorrow.

For your information, Iÿgot a dev/git-based installation working where Iÿcan easily divert the haproxy traffic to my dev machine with git and all. I could reproduce the issue, so I think Iÿam ready to try whenever you are too.

[echicken]

To: martylake

Re: Re: synchronet behind a reverse proxy

By: martylake to echicken on Thu Nov 25 2021 22:14:53

ma> For your information, Iÿgot a dev/git-based installation working where
ma> Iÿcan easily divert the haproxy traffic to my dev machine with git and
ma> all. I could reproduce the issue, so I think Iÿam ready to try whenever
ma> you are too.

Okay, if you want to switch to the 'websocket-haproxy' branch and pull, you'll have my changes to:

exec/load/ftelnethelper.js
exec/websocketservice.js

Shouldn't be necessary to restart anything.

If I magically got everything right on the first try, then that should do it - but I'm not optimistic that's going to happen. At least we've got something to start with.

---
echicken
electronic chicken bbs - bbs.electronicchicken.com
---

þ Synchronet þ electronic chicken bbs - bbs.electronicchicken.com

[martylake]

To: echicken

Re: Re: synchronet behind a reverse proxy

By: echicken to martylake on Fri Nov 26 2021 06:50 am

> If I magically got everything right on the first try, then that should do it
> - but I'm not optimistic that's going to happen. At least we've got
> something to start with.

I tested your implementation, it almost worked, I had to add padding so that the address length in bytes occupates two bytes (15th and 16th), instead of one, as indicated by the spec.

Can I paste a git patch in SyncTERM ? Is it proper etiquette ?

-+-
exec/websocketservice.js | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/exec/websocketservice.js b/exec/websocketservice.js
index 08bdbb7a1..25b166f71 100644
-+- a/exec/websocketservice.js
+++ b/exec/websocketservice.js
@@ -95,9 +95,9 @@ try {
if (UsingHAProxy()) {
var hapstr = '\x0D\x0A\x0D\x0A\x00\x0D\x0A\x51\x55\x49\x54\x0A\x21';
if (client.socket.family === PF_INET) {
- hapstr += '\x11\x0C';
+ hapstr += '\x11\x00\x0C';
} else if (client.socket.family === PF_INET6) {
- hapstr += '\x21\x24';
+ hapstr += '\x21\x00\x24';
}
hapstr += inet_pton(client.ip_address);
hapstr += inet_pton(FServerSocket.remote_ip_address);
--
2.30.2

[echicken]

To: martylake

Re: Re: synchronet behind a reverse proxy

By: martylake to echicken on Thu Nov 25 2021 23:44:08

ma> I tested your implementation, it almost worked, I had to add padding so
ma> that the address length in bytes occupates two bytes (15th and 16th),
ma> instead of one, as indicated by the spec.

Good catch - thank you.

ma> Can I paste a git patch in SyncTERM ? Is it proper etiquette ?

I have no problem with it, especially if it's well-formatted for the medium (80x24).

I just pushed a couple of changes: your patch, and a fix because the websocket service was sending the wrong client address. Please update and let me know how it goes.

---
echicken
electronic chicken bbs - bbs.electronicchicken.com
---

ş Synchronet ş electronic chicken bbs - bbs.electronicchicken.com

[echicken]

To: martylake

Re: Re: synchronet behind a reverse proxy

By: martylake to echicken on Thu Nov 25 2021 20:19:04

ma> When I get this whole setup working, I can suggest a few edits to
ma> synchronet's wiki's proxy page to account for all those peculiarities.

I forgot to mention: you don't need to suggest edits. You can just go ahead and make them. Anyone with a QWK account on Vertrauen can log into the wiki.

It'd be best if people who have a stake in the HAProxy support were active editors of relevant pages. I don't use it and only have a cursory understanding at the moment.

---
echicken
electronic chicken bbs - bbs.electronicchicken.com
---
ş Synchronet ş electronic chicken bbs - bbs.electronicchicken.com

[martylake]

To: echicken

Re: Re: synchronet behind a reverse proxy

By: echicken to martylake on Fri Nov 26 2021 01:46 pm

> Good catch - thank you.

You are welcome.

> I have no problem with it, especially if it's well-formatted for the medium
> (80x24).

Duely noted.

> I just pushed a couple of changes: your patch, and a fix because the
> websocket service was sending the wrong client address. Please update and
> let me know how it goes.

I just pulled, and I confirm it works great for me. I did not a difference for the address fix, as Iĸuse the same computer for dev and testing. I looks good to me though.

What is the process to deliver this fix to the docker image ? And how do you manage synchronet's configuration files with regard to upgrading synchronet ? The way I did it (modifying the files in ctrl/ and overwriting /web from /backups), I feel like I will loose these modifications next time I pull, but maybe I am wrong ?



I have an unrelated issue, should I open a new "thread" for it ? YMODEM-G upload with the ftelnet ws works for very small files (<144ko) but does not work anymore for bigger files (>174ko).

Here are the logs. I don't know how to paste from syncterm, but I will type them:
* sexyz: !Receive timeout (3 seconds)
this one several times while Iĸchoose the menu upload and selets my file in the OS file picker
* sexyz: !Too many errors (1)
* sexyz: !File Transfer Failure
* sexyz: !Error fetching YMODEM header block
While this happens, ftelnet sends all the file, and synchronet treats them as inputed keys.

I don't reproduce the issue if I upload files using syncterm.

Is it a known defect ?

Best,

[echicken]

To: martylake

Re: Re: synchronet behind a reverse proxy

By: martylake to echicken on Fri Nov 26 2021 13:54:55

ma> I just pulled, and I confirm it works great for me. I did not a difference
ma> for the address fix, as Iÿuse the same computer for dev and testing. I
ma> looks good to me though.

Glad to hear that it's working.

A few other items before we're done with this one:

1) Currently, the websocket service will terminate if the HAPROXY_PROTO option is set, but the X-Forwarded-For header is absent. It won't connect somebody to your terminal server if it can't send their "real" IP address.

This wouldn't work for someone who was using HAProxy in front of their terminal server, but wasn't using an HTTP reverse proxy. Seems an unlikely scenario, but I thought I'd raise it anyway.

We could have it fall back to sending the address of the websocket server (ie. your BBS), or some dummy value if the X-Forwarded-For header is absent. This would allow the websocket connection to proceed, but would result in an incorrect client address.

2) A bit of testing before I merge this:

2a) If you remove the HAPROXY_PROTO option in sbbs.ini, then restart, do websocket connections (ftelnet) continue to work?

2b) If you remove the HAPROXY_PROTO option in sbbs.ini, restart, and also remove the HTTP reverse proxy from the mix (remember to adjust/remove wsp and wssp in modopts.ini->[web] also) do websocket connections continue to work?

2c) If you leave HAPROXY_PROTO enabled, but remove the HTTP reverse proxy from the mix, do websocket connections fail?


I'll reply to your other comments in another message.

---
echicken
electronic chicken bbs - bbs.electronicchicken.com
---
ş Synchronet ş electronic chicken bbs - bbs.electronicchicken.com

[echicken]

To: martylake

Re: Re: synchronet behind a reverse proxy

By: martylake to echicken on Fri Nov 26 2021 13:54:55

ma> What is the process to deliver this fix to the docker image ? And how do

When we're satisfied with how it's all set up and testing is done, I'll merge this branch into the main one. Anyone who updates or installs Synchronet afterward will get the changes.

However your docker-based install normally gets updated, that'll take care of it.

(Allow that these changes won't be in nightlies immediately.)

ma> you manage synchronet's configuration files with regard to upgrading
ma> synchronet ? The way I did it (modifying the files in ctrl/ and
ma> overwriting /web from /backups), I feel like I will loose these
ma> modifications next time I pull, but maybe I am wrong ?

I don't know enough about how your docker image is set up / updated to say for sure. If it's like this:

http://wiki.synchro.net/install:docker

It's unclear to me whether the stuff listed under "Volumes" is impacted when you update to a newer image.

They single out ctrl/text.dat as something that gets overwritten if there's an updated version; this is generally unnecessary, and is unnecessarily destructive.

Meanwhile I'm not sure what the deal is with the /sbbs/web directory. I get the sense that you're meant to update it manually as you go. (There are probably better ways this could be set up, for webv4 at least.)

Maybe someone more familiar with this setup can chime in. That wiki page needs some building out.

ma> I have an unrelated issue, should I open a new "thread" for it ? YMODEM-G
ma> upload with the ftelnet ws works for very small files (<144ko) but does
ma> not work anymore for bigger files (>174ko).

ma> I don't reproduce the issue if I upload files using syncterm.

It's *most likely* a problem with fTelnet, but it's not impossible that both SEXYZ and SyncTERM are doing something wrong in such a way as to be compatible with each other.

I'd start by creating an issue in the fTelnet project on GitHub (if that's where he still hosts it).

You could create an issue on Synchronet's GitLab server as well, but as I said, it's less likely to be a problem on that side.

Short of testing fTelnet with some other software (eg. Mystic) or with Synchronet using a different handler for YMODEM-G (scfg->File Options->Transfer Protocols), I can't think of anything else you can do to narrow it down.

---
echicken
electronic chicken bbs - bbs.electronicchicken.com
---
ş Synchronet ş electronic chicken bbs - bbs.electronicchicken.com

[martylake]

To: echicken

> 1) Currently, the websocket service will terminate if the HAPROXY_PROTO
> option is set, but the X-Forwarded-For header is absent. It won't connect
> somebody to your terminal server if it can't send their "real" IP address.

as per http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
```
The receiver MUST be configured to only receive the protocol described in this
specification and MUST not try to guess whether the protocol header is present
or not. This means that the protocol explicitly prevents port sharing between
public and private access. Otherwise it would open a major security breach by
allowing untrusted parties to spoof their connection addresses. The receiver
SHOULD ensure proper access filtering so that only trusted proxies are allowed
to use this protocol.
```

So I think that
> the websocket service terminating if the HAPROXY PROTO option is set and the
> X-Forwarded-For header is absent
really is the *expected and correct* behavior.

Any other workaround such as replacing the client IP with the websocket proxy
IP would have undesirable side effects, like having the websocket proxy banned
because of multiple login tentatives or script kiddies.

> 2) A bit of testing before I merge this:

> 2a) If you remove the HAPROXY_PROTO option in sbbs.ini, then restart, do
> websocket connections (ftelnet) continue to work?

I tested, and it continues to work.

> 2b) If you remove the HAPROXY_PROTO option in sbbs.ini, restart, and also
> remove the HTTP reverse proxy from the mix (remember to adjust/remove wsp
> and wssp in modopts.ini->[web] also) do websocket connections continue to
> work?

So, instead of connecting at forum.talbot.audio, I connect at localhost:8880
(overriden port for http). Is this equivalent to (remove the HTTP reverse
proxy) ?

The websocket connection continues to work.

> 2c) If you leave HAPROXY_PROTO enabled, but remove the HTTP reverse proxy
> from the mix, do websocket connections fail?

The websocket connection stops working because the header is absent. As per
question 1), I think that the behavior is correct and to be expected.
Here is this message I got:
11/27 08:45:04 srvc 0029 WS Error: BBS is using HAProxy, but no X-Forwarded-For
header present.

[echicken]

To: martylake
Re: Re: synchronet behind a reverse proxy

By: martylake to echicken on Sat Nov 27 2021 00:53:34

ma> The receiver MUST be configured to only receive the protocol described in
ma> this specification and MUST not try to guess whether the protocol header
ma> is present or not. This means that the protocol explicitly prevents port

Technically that's a different scenario. The websocket server would always send the PROXY header, it just wouldn't always reflect the "true" client IP address.

ma> So I think that

>> the websocket service terminating if the HAPROXY PROTO option is set
>> and the X-Forwarded-For header is absent

ma> really is the *expected and correct* behavior.

I agree. Just wanted someone else to agree with me before I made it the default behaviour. We can always adjust later.

ma> So, instead of connecting at forum.talbot.audio, I connect at
ma> localhost:8880 (overriden port for http). Is this equivalent to (remove
ma> the HTTP reverse proxy) ?

Yep, if localhost:8880 is what your BBS webserver actually listens on, and is what the reverse proxy treats as upstream, then it's the same effect.

Okay, so it sounds like various cases are being handled as expected. I'll merge this in later today.

Thanks for your help with testing all of that - I made these changes without actually running any of the scripts myself.

---
echicken
electronic chicken bbs - bbs.electronicchicken.com
---
ş Synchronet ş electronic chicken bbs - bbs.electronicchicken.com

[martylake]

To: echicken

Re: Re: synchronet behind a reverse proxy

By: echicken to martylake on Sat Nov 27 2021 03:06 pm

> Okay, so it sounds like various cases are being handled as expected. I'll
> merge this in later today.

Sounds good to me !
\o/

> Thanks for your help with testing all of that - I made these changes without
> actually running any of the scripts myself.

Thanks for implementing it so fast, it was a pleasure working with you.

[Tracker1]

To: martylake
On 11/24/21 12:52, martylake wrote:
>
>> Is your syncrhonet running on Windows by chance?
>
> Unfortunately not. It's running on raspberrypi4 raspbian under docker.

Are you running the bbs-io image, or something you cooked up?

Only asking as I was considering changing on how the volumes are done...
instead of having each subdirectory under sbbs mounted, simply mounting
/sbbs-files and the "init" at startup would symlink from /sbbs (this way
/exec in particular is still preserved/original and the rest would
largely work the same.
--
Michael J. Ryan - trac...@roughneckbbs.com
---
￭ Synchronet ￭ Roughneck BBS - roughneckbbs.com

[Tracker1]

To: martylake

On 11/25/21 14:04, martylake wrote:
>> Sorry, I should have asked if you did a development / git-based installation
>> in the first place. Once you have one working, let me know. I'll tinker with
>> this a bit tonight if I'm able, or tomorrow.
>
> Ok, I will focus on getting a git-based installation ready ASAP and let you
> know. Thank you very much for offering help on that !

The daily builds of the bbs-io/synchronet image is from git... anything
committed should arrive as a new nightly within 48 hours.

nightly-20211201
Last pushed 14 hours ago by tracker1

https://hub.docker.com/repository/docker/bbsio/synchronet/tags

--
Michael J. Ryan - trac...@roughneckbbs.com
---
￭ Synchronet ￭ Roughneck BBS - roughneckbbs.com

[Tracker1]

To: deon

On 11/25/21 16:38, deon wrote:
>>> So reconfigure the websockets part to connect to haproxy's port,
>>> not SBBS directly.
>>
>> The issue is that websocket is a protocol on top of http, that's
>> why we need a websocket->telnet proxy. The issue if I understand
>> correctly is that this proxy is not forwarding haproxy information
>> to the backend.
>
> Correct, there are two paths that could address this.
>
> * The websocket connects to haproxy, not SBBS directly.
>
> * The websocket speaks that haproxy protocol when talking with
> SBBS directly.
>
> The first is probably easy to do, if you can configure it via a
> config file (sorry I dont know).
>
> The second would require development I'm guessing...

The haproxy websocket connection should forward the given path(s) to the
websocket port on synchronet as a tcp connection... or upcycle... I
don't know if haproxy supports acting as a websocket<->tcp gateway directly.

I've tried to make a specific path or websocket connection upgrade work
to reverse-proxy to a different port, but it doesn't want to work
correctly... kind of wish that websocket connection
support/configuration was directly in the sbbs http service itself,
which would make that part much easier to work with.

--
Michael J. Ryan - trac...@roughneckbbs.com
---
￭ Synchronet ￭ Roughneck BBS - roughneckbbs.com

[Tracker1]

To: echicken

On 11/27/21 13:06, echicken wrote:
> Okay, so it sounds like various cases are being handled as
> expected. I'll merge this in later today.
>
> Thanks for your help with testing all of that - I made these
> changes without actually running any of the scripts myself.

Sorry for jumping in late on this one... was just catching up on messages.

--
Michael J. Ryan - trac...@roughneckbbs.com
---
￭ Synchronet ￭ Roughneck BBS - roughneckbbs.com

[Tracker1]

To: echicken

On 11/26/21 02:08, echicken wrote:
>
> You mentioned something in an earlier message about HAProxy
> lowercasing the names of HTTP headers. If the client sends 'Some-
> Header: Some Data', can I expect the upstream to receive 'some-header:
> Some Data'? This will become important, because it looks like the
> websocket service currently does case-sensitive matching on header
> lines.

The name portion of headers is supposed to be case-insensitive for HTTP.
And from the prior message, that's my understanding is that it is
indeed normalizing to lower-case.

--
Michael J. Ryan - trac...@roughneckbbs.com
---
￭ Synchronet ￭ Roughneck BBS - roughneckbbs.com

[martylake]

To: Tracker1

Re: Re: synchronet behind a reverse proxy

By: Tracker1 to martylake on Wed Dec 01 2021 11:17 pm

Dear Tracker|

> Are you running the bbs-io image, or something you cooked up?

Yes, I am running the nightly from docker hub.

> Only asking as I was considering changing on how the volumes are done...
> instead of having each subdirectory under sbbs mounted, simply mounting
> /sbbs-files and the "init" at startup would symlink from /sbbs (this way
> /exec in particular is still preserved/original and the rest would
> largely work the same.

It sounds like a good idea to me, even if I don't understand what would be the benefits apart from saving some lines of volumes mount in docker-compose.yml ?