Salt over the internet

[Ed Killian]

We have been using salt to provide configuration management within our networks. But now management wants to start using it for our offsite locations and company systems that are located in employee's WFH locations. I'm extremely hesitant to expose salt to the internet but they want to be able to manage these systems even when they aren't connected to our VPN. Is there a safe way to do this or do we need to start looking for a different solution?

Thanks.

[Nerigal]

Hi,

 

i don't think im aware enough of your topology but you shouldn't have to expose the salt-master to the universe... by this, i mean that you could allow 4505 and 4506 in your firewall to - from your office subnet to the salt-master external ip

or to the nat ip from your office, usually they do know
the requirement would be to have a static IP from minion side as well as the master
so you don't have to manipulate firewall rules every now then.

If im not wrong, by default salt encrypt the communication between the master and minion using an AES secure key that is rotated each time the salt-master is restarted and / or the minion key is deleted from the salt-key command

other than that you can upgrade the encryption using this param in the master config and the minions as well

# Use TLS/SSL encrypted connection between master and minion.
# Can be set to a dictionary containing keyword arguments corresponding to Python's
# 'ssl.wrap_socket' method.
# Default is None.
#ssl:
#    keyfile: <path_to_keyfile>
#    certfile: <path_to_certfile>
#    ssl_version: PROTOCOL_TLSv1_2

There surely are more hardening options to be considered but out of my head that what i would suggest to look into

Thank you
Nerigal

On 2021-06-23 11:45, Ed Killian wrote:

We have been using salt to provide configuration management within our networks. But now management wants to start using it for our offsite locations and company systems that are located in employee's WFH locations. I'm extremely hesitant to expose salt to the internet but they want to be able to manage these systems even when they aren't connected to our VPN. Is there a safe way to do this or do we need to start looking for a different solution?

 

Thanks.

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/5528c83a-d6b6-4d7d-a8eb-08c8fab8be7an%40googlegroups.com.

[Ed Killian]

To clarify:

[salt-master] ------- [firewall] ------- {internet}  ----------- [minion]

The problem with what you propose is that we can't guarantee that the minion's IP won't change. My home IP has changed several times in the last few years because of changes the ISP has made. And therefore we would need to expose port 4505 and 4506 to the world.

[Dafydd Jones (techneg.it)]

Hi,

in the #security channel of the Salt Community Slack a user has written a document called "Leveraging SaltStack as Part of NetFoundry NaaS" which afaiui is a commercial zero trust networking platform.

It could give you some ideas.

HTH,

Dafydd

On Wed, 23 Jun 2021 at 16:45, Ed Killian <edtech...@gmail.com> wrote:

We have been using salt to provide configuration management within our networks. But now management wants to start using it for our offsite locations and company systems that are located in employee's WFH locations. I'm extremely hesitant to expose salt to the internet but they want to be able to manage these systems even when they aren't connected to our VPN. Is there a safe way to do this or do we need to start looking for a different solution?

Thanks.

--

[Nerigal]

and is it possible for you to ask your ISP to be on a static ip for your office / home ?

 

Nerigal

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/67a2588a-8bcd-443a-adbe-2c19ebb9045an%40googlegroups.com.

[brent timothy saner]

On 6/23/21 17:17, Nerigal wrote:
> Hi,
>
> i don't think im aware enough of your topology but you shouldn't have to
> expose the salt-master to the universe... by this, i mean that you could
> allow 4505 and 4506 in your firewall to - from your office subnet to the
> salt-master external ip

OP mentioned WFH minions, which immediately made me think of dynamic WAN
IP addresses. Salt, being a agent-based engine (as opposed to, for
instance, Ansible), this isn't an issue. However, it does require
opening port 4505/tcp and 4506/tcp on the master, yes.

There are various ways to address this concern:

- A "sign-in" portal for employees that, upon successful login, grabs
their IP and adds it to an address whitelist in the firewall (Much as AI
is, essentially, just a bunch of if-tests, this is exactly what
"zero-trust" networks boil down to - along with credential/identity
verification, of course.)

- A VPN with pushed selective routing; only a route for the master is
pushed to the client.
-- Despite the hype around "zero-trust" networks and them being the
current hotness, VPNs really do offer some benefits zero-trust does not.
It allows you to run things like WINS in an encrypted tunnel over the
WAN, which can be immensely useful, and gives a much more "in-office"
feel to your network topology for WFH architexture, so do not discount
it immediately.

However, these may not even be necessary depending on your risk model.
Continue reading below.

>
> or to the nat ip from your office, usually they do know
> the requirement would be to have a static IP from minion side as well as
> the master
> so you don't have to manipulate firewall rules every now then.

Most residential ISPs do not offer this; or, if they do, it's a
typically significantly higher cost tied to a plan upgrade. This is
something your company would then need to sink as a cost (as it'd be a
compensation per each employee). An unnecessary cost, at that, because
again - static IP addresses are not necessary for agent-driven
configuration management engines.

>
> If im not wrong, by default salt encrypt the communication between the
> master and minion using an AES secure key that is rotated each time the
> salt-master is restarted and / or the minion key is deleted from the
> salt-key command

This is somewhat accurate. From what I've been able to determine through
poking around:

- The master key does not rotate as the minions need the public key to
establish a proper encryption session. It can be changed/manually
rotated, but this is... not advised because if you have master key
fingerprint verification[0] enabled, the minion will refuse to connect
to the master.
Now, there is a *session* key that IS rotated. But this is ephemeral,
not the actual master key, and is encrypted to the minion's public key.
As long as your minions aren't sharing private keys (and they shouldn't,
because a LOT more will break in addition to encryption), each session
is encrypted to only that minion <=> master interaction. See [1] for
details.
(Sidenote: this can be disabled, if desired. via rotate_aes_key[2].)

- I don't believe (though am certainly welcome to correction) that there
even *is* a way to run a master <=> minion communication that is *not*
encrypted. In other words, not only is it encrypted by default but it
*must* be encrypted by design.

- AES is indeed used. I'm not sure if it's AES128-CBC or AES256-CBC, but
either is considered very, very safe and resistant.

> # Use TLS/SSL encrypted connection between master and minion.
> # Can be set to a dictionary containing keyword arguments corresponding
> to Python's
> # 'ssl.wrap_socket' method.
> # Default is None.
> #ssl:
> #    keyfile: <path_to_keyfile>
> #    certfile: <path_to_certfile>
> #    ssl_version: PROTOCOL_TLSv1_2
>

This[3] has to do with the TLS session itself, which is quite a bit more
higher-level than the actual encryption algorithm used.

> There surely are more hardening options to be considered but out of my
> head that what i would suggest to look into

There is existing documentation[4] for this.

Now, all this to say that, in *theory*, it is perfectly and absolutely
safe to expose a salt master's 4505/tcp and 4506/tcp to the WAN
(0.0.0.0/0, ::/0).

However, in *practice*, judiciousness is recommended. There have been
instances where the encryption itself may be solid, but the
*implementation* was flawed. This has occurred within SaltStack itself
before[5]. Thankfully, SaltStack (both SaltStack Project and its
community contributors) are more active in addressing these concerns
than the average project typically, so patches are in place quite
commonly very quickly once discovered, but this *does* require the
flexibility to be able to update the master and minion daemons to fix
these issues quite commonly.

As always, security is quantitative and qualitative, not binary. You are
not "secure" or "insecure", you are "more secure" or "less secure". This
is not only determined by your implementation but highly dependent on
your risk and threat model - and that is something this list cannot
determine for you.

TL;DR: "You're probably fine running it open on the WAN, but caveat
emptor and I'm not liable for damages."


[0]
https://docs.saltproject.io/en/latest/ref/configuration/index.html#master-key-fingerprint
[1] https://docs.saltproject.io/en/getstarted/system/communication.html
[2]
https://docs.saltproject.io/en/latest/ref/configuration/master.html#rotate-aes-key
[3] https://docs.saltproject.io/en/latest/ref/configuration/master.html#ssl
[4] https://docs.saltproject.io/en/latest/topics/hardening.html
[5] https://saltproject.io/security_announcements/

https://www.cvedetails.com/vulnerability-list/vendor_id-12943/product_id-26420/Saltstack-Salt.html
https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=SaltStack

[Grant Haywood]

Don't expose your 4505 or 4506 without firewall ACL's, there have been critical vulnerabilities that allow salt-master to be compromised as recently as last year.

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/6bacf675-79b7-de23-33dc-2371646cd902%40gmail.com.

[Christopher Hawkins]

You can change the ports (or set up a new salt master with them if existing server is a problem to reconfigure) to a random high number like 45000 or something. Those never get scanned and will reduce your exposure to almost nothing. 

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/CAPayrd5deDxDCxT%2BBqqKzLyOpzAskz1%3DvScO34rgqPNFHKpZow%40mail.gmail.com.

[brent timothy saner]

On 6/26/21 18:35, Grant Haywood wrote:
> Don't expose your 4505 or 4506 without firewall ACL's, there have been
> critical vulnerabilities that allow salt-master to be compromised as
> recently as last year.

I addressed this in my previous reply. See reference #5 and the
paragraph starting with "However, in *practice*, judiciousness is
recommended. ..."


It is worth noting that there have been plenty of vulnerabilities in
iptables[0], even Cisco's IOS[1], etc. A firewall is but one layer in
comprehensive security, one that is subjective to its own failures, and
sometimes (depending on your implementation and risk) it's perfectly
fine to leave it open to WAN. Keep in mind that Salt is 10 years old,
and to my knowledge that slew of vulnerabilities reported were the only
ones that led to something like this.

In some cases, firewalling is simply not an option without other support
infrastructure which may not be possible.

This is why risk and scope must be considered instead of "always" and
"never". Is a firewalled salt master *MORE* secure than an unfirewalled
one (assuming proper firewall configuration)? Absolutely. Is it a death
sentence if you have your master open" *Absolutely not.* There's much
lower-hanging fruit to be concerned about, like open password-auth SSH
login. (Have you secured your in-house git servers today? etc.)




[0] https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=iptables
[1] https://www.cvedetails.com/product/19/Cisco-IOS.html?vendor_id=16

[brent timothy saner]

On 6/26/21 18:42, Christopher Hawkins wrote:
> You can change the ports (or set up a new salt master with them if
> existing server is a problem to reconfigure) to a random high number
> like 45000 or something. Those never get scanned and will reduce your
> exposure to almost nothing. 

Personally, I always scan every single port, 1-65535[0], when I'm
mapping ports - with service detection. It usually takes less than 30
minutes. More towards the 2-5 minute range if there aren't a lot of
services actually running.

While using a non-standard port may prevent noisy logs, that's about all
it affords you. If you are a target, it will offer nothing.


[0] For those unaware, 0 is reserved for dynamic allocation by the
stack. Don't bother scanning it. :P

[Andreas Thienemann]

Hi Ed

We have been using salt to provide configuration management within our networks. But now management wants to start using it for our offsite locations and company systems that are located in employee's WFH locations. I'm extremely hesitant to expose salt to the internet but they want to be able to manage these systems even when they aren't connected to our VPN. 

I’d say you are correct in being hesitant. In theory opening up the salt master to the internet _should_ be risk free as all connections are authenticated and encrypted.

In practice however we have seen that pre-auth-bypass exploits exist and just in the past two years there have been nasty bugs found in SaltStack that led to existing masters being taken over for crypto-mining.

Is there a safe way to do this or do we need to start looking for a different solution?

I believe nearly _every_ other solution has similar theoretical problems. HTTP based MDM solutions might be better because while the application itself might be vulnerable, at least the Webservers they are running on are usually well understood and can be hardened easily.

But for SaltStack what I would suggest would be the following approach:

Option A: Use a VPN. It doesn’t have to be the same VPN your users use to connect to the company, but something that authenticates access to the master and is happening on a layer below the application.

Option B: Wrap the connections to the salt-master in a client authenticated TLS connection. This way your clients will present a signed client certificate that authenticates them and _only_ if this authentication succeeds will the ZeroMQ packets be forwarded to the salt-master.

You can build such a setup relatively easy using stunnel:

salt-master <—zeroMQ —> stunnel <— zeroMQ over TLS over Internet—> stunnel on minion <— zeroMQ —> salt-minion

A simple configuration HOWTO using pre-shared-keys (which is a less secure alternative than per client certificates as everyone is using the same key) can be found at https://www.stunnel.org/auth.html.

In such a setup your minion will not contact your master directly but instead it will connect to localhost on port 4505 and 4506. Stunnel is listening on both these ports and is forwarding those connections to an internet-exposed host at your company wrapping the date in TLS packets.

That internet connected host has another two stunnel instances listening on port 4505 and 4506 and will forward those ports using the "connect = salt-master:4505“ (4506 respectively) configuration setting.

Good luck,

 Andreas

[viq]

I've been looking some at that, and while I didn't yet set it up, from
my reading my vote goes to tailscale combined with
https://github.com/juanfont/headscale - especially with preauth keys
it's very simple to set up, doesn't try to grab all the traffic (unless
you want it it), and allows to reach a specific service over an
encrypted tunnel almost regardless of local network conditions.
--
viq