Looking for a specialized proxy package

[Michael Ellis]

I posted a question about this on ServerFault last week but didn't get any answers other than a few comments from one person who said (basically) "use a VPN".   That seems like overkill.  I'm trying to find a reliable way to proxy occasional HTTP access to any of  ~100 geographically dispersed IOT devices through a cloud server.  

I'm using Go on the cloud server and on the IOT devices, so I thought I'd ask here.

Situation:

• We have complete control over the configuration of the IOT devices and the cloud host.
• We don't have control of the customers' routers and firewalls, but can specify minimum requirements for port openings, etc.
• FWIW, the IOT devices are BeagleBone Black running Debian Buster and the cloud host will be, typically, a multi-core droplet (or similar) running Linux.
• The IOT's serve dynamic web pages over HTTP. (HTTPS doesn't seem feasible because of certificate requirements and overall load on the IOT cpu.) The cloud host will have HTTPS capability.
• This is a low-traffic situation. The IOT's report some overall status information (via rsync/ssh) at 4 minute intervals). We already have a web interface (written in Go) on the cloud server that aggregates and displays the status reports.
• Access to an IOT's web service will only occur when a user wants to investigate a problem report in more detail. Typically, only one or two users will have credentials to browse the cloud server.

The scheme I have in mind is: 

1. At configuration time for each IOT device the installation tech will use ssh-copy-id to install the IOT device's public key on the cloud service.
2. The IOT device will  then remotely execute a one-shot program (already written and tested) on the cloud server.  The IOT will provide a unique identifier as an argument and the program will return a permanent port number and add a record to a database to record the assignment.
3. The IOT will open a reverse SSH tunnel on the server (probably managed by auto-ssh) specifying the permanent port on the server and a local port on which it will listen for HTTP requests.
4. The cloud server, when generating status report pages, will include a link to fetch the home page of each IOT device by embedding its unique identifier specified in step 2 above.

The piece I'm missing is how to construct a proxying handler that will use the identifier in the link to look up the tunnel port and fetch the IOT's home page and thereafter make it seem as though the user is directly browsing the IOT.

Any help appreciated (and thanks for reading this far!)

[Michael Ellis]

FWIW,  I pasted my  post into ChatGPT-4 and got what might be a plausible outline of an approach using httputil.NewSingleHostReverseProxy.

But, as we know, LLM's are prone to hallucination. If you're curious, here's a share link.

https://shareg.pt/cNoNdWc

[Matthew Zimmerman]

Honestly I'd probably use grpc and keep a constant connection from the IOT to the cloud.  No ports/services required on the client at all and the server can still request things in real time.

Like: https://www.talentica.com/blogs/part-3-building-a-bidirectional-streaming-grpc-service-using-golang/

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/4eee9142-17f1-4d78-9057-9702e1d2d557n%40googlegroups.com.

[Michael Ellis]

Thanks, Matthew.  I know what RPC is, but have never considered it as a way to serve http from behind a NAT.  I should say that the IOT's are part of a product that's been in the market for several years.  My client likes the web interface we built and wants users to be able to access an IOT's pages through a secure intermediary server.  How would that work with grpc?

[Vladimir Varankin]

Hey Michael,

> The piece I'm missing is how to construct a proxying handler that will use the identifier in the link to look up the tunnel port and fetch the IOT's home page and thereafter make it seem as though the user is directly browsing the IOT.

If I got the question right, there are two parts:

1. Build an HTTP reverse proxy, which routes the requests to a target device's server (i.e. the upstream).

Go's httputil.ReverseProxy (https://pkg.go.dev/net/http/httputil#ReverseProxy) can do that. You will need to implement httputil.ReverseProxy.Rewrite to route a request.

2. Use the identifier in the link to look up the tunnel port

Not sure, if I've missed what "link" means in this context — is this a network interface, the devices are connected to?

[Tamás Gulácsi]

As far as I understand, the requirement is to have a link presented by the cloud server, that is proxied to the IOT's HTTP handler.

The latter is only on the IOT's localhost, accessible on the cloud server through ssh reverse tunneling (so, a specific port on the cloud server's localhost).

This is easy: just have a handler on the cloud server (say, /p/<unique-id-of-the-IOT>) that

1. looks up the unique id and fetches the proper http://localhost:permanent-port.

2. The hard part is that you have to rewrite each URL in each response, otherwise the links and imports won't work.

All this can be achieved with net/http/httputil#ReversProxy, with proper Rewrite (for 1.) and ModifyResponse (for 2.) methods.

2. may be elided by putting the IOT under a subdomain on the server  - but this needs * SSH certificate (or Let'sEncrypt  - Caddy may help).

GT

[egon...@eldondev.com]

I've seen this type of SSH/rsync access used many times (and have used
it myself to great effect), but it is worth considering if you feel
confident being able to sufficiently secure the ssh access in your
threat model. Your system, if compromised, could serve as an ingress and
pivot to other systems within your customer's perimeter. I would say
that is why you might get recommendations to avoid SSH altogether and
use something like grpc or MQTT, and to limit the scope of the lateral
access through the systems. I think it's worth worrying about that, and
using a vpn and issuing per-device certificates is worthwhile, but I am
also painfully aware of the ins and outs of managing those parts of the
system. If you're not particularly worried about that part of the threat
model, or you feel like you have all the bases covered, I think your
ssh/proxy strategy would work, probably even with the reverse proxy
utility you mentioned, although I'm not sure how well that specific
component supports the user access control you might need.

Eldon

[Matthew Zimmerman]

I would abandon the IOT device doing any serving of data to any client and instead make your cloud instance the web server rendering data it obtains via grpc from the IOT devices.  This makes the design much simpler on the IOT side and takes any direct access to them away which simplifies the design and security.  With proxying traffic, you seem to essentially want to do that at the network level anyway, why not go further up the stack?

If the IOT device knows only how to answer specific requests for information (and has no ability to listen for outside network traffic) the surface area for attack of that device is made very small.

e.g., client-http-cloud-grpc-iot-grpc-cloud-http-client 

One network connection from cloud to IOT, not multiple.  

Specific requests being able to be handled by the IOT device, not generic ssh/rsync/http.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/8f4ac8b1-4dc8-4abf-8b7b-608b0ef3886an%40googlegroups.com.