Configuring SMTP in the docker image

[Matthew Saltzman]

I am trying to set up email via SMTP so users can get password resets. In my organization, all email must be sent via a smarthost. I haven't found any docs explaining how the SMTP service works in the CoCalc Docker image or how to configure it.

Can anyone here help? Any suggestion where else to look?

TIA.

[hsy]

Hi, are you an "admin"?

If not: https://github.com/sagemathinc/cocalc-docker#make-a-user-an-admin

With that enabled, you get an "Admin" tab in the CoCalc main app (in "/admin") and there you have to enable email and enter SMTP settings.

Afterwards, restart the server to activate the settings.

-- Harald

[Matthew Saltzman]

I am an admin. I see where I can select SMTP as the mailer service back end, but I need to set the smart host IP address, and I don't see any place on the settings page to make that happen.

[Harald Schilly]

Hello Matthew,

I assumed this is just a regular IP address or domain name. I don't know what a "smart host IP address" is. Can you point me in the right direction please? 

Under the hood, these settings are used to configure node's nodemailer:

https://nodemailer.com/smtp/#1-single-connection

Maybe nodemailer does not support this in the first place?

-- Harald

--
CoCalc: Collaborative Calculation in the Cloud -- https://cocalc.com
---
You received this message because you are subscribed to a topic in the Google Groups "CoCalc" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cocalc/_mFdNUewxXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cocalc+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cocalc/8ab30fb5-86ac-4246-a8c6-fce648b79074n%40googlegroups.com.

[Matthew Saltzman]

"Smart host" is Sendmail's term for a relay target that your system sends all outbound mail through. So I want nodemailer to just route all outbound mail to my organization's mail server.

It looks like that transport code is where nodemailer gets the information to create the connection, but I wonder if there's someplace in CoCalc where those settings are configured. I'd rather have the caller ask for the right thing than hack nodemailer's code to do it.

[Matthew Saltzman]

After some hunting, it looks like I need to set the environment variable COCALC_SETTING_EMAIL_SMTP_SERVER. So I think my question reduces to: Where do I set environment variables for configuring the server on startup?

[Harald Schilly]

All those environment variables are just a way to set the admin server settings during startup (without touching the user interface, so that it can be integrated with your deployment configuration).

They're parsed and saved in the database in the "server_settings" table.

You can set values in that postgres table manually as well.

In particular, that setting is "SMTP server (for email)", under "Email Configuration", when the backend type is "smtp".

Below is a screenshot for reference. It's all set by those environment variables, hence it is read-only and disabled. If you do not set those environment variables, you can enter all those values right there manually.

In any case, after any changes there, and they're saved and stored in that table in the database, you have to restart cocalc's services to make those changes effective.

-- Harald

To view this discussion on the web visit https://groups.google.com/d/msgid/cocalc/eb3b4075-c872-4240-919b-f12a360bc220n%40googlegroups.com.

[Matthew Saltzman]

OK Not seeing where to configure the mail host in the settings page was due to a stupid mistake on my part. I solved that, so now I can select SMTP mail and see the place to fill in hostname, etc.

I can send mail to the host from inside docker using telnet and typing the actual SENDMAIL dialog commands HELO, EHLO, MAIL, RCPT, DATA, etc.), but sending mail from the CoCalc Admin page fails.

[Matthew Saltzman]

Progress, but still no joy.

My organization's relay server doesn't support AUTH, but apparently, nodemailer insists on it. So I installed Postfix on my container server and configured it to use SASL, listen on port 587 on the container's network, and forward mail to my organization's relay. (The container is 172.17.0.2 and the host is 172.17.0.1.) Now I can connect to Postfix with telnet from the container's shell and send mail by hand, but sending a password reset test from CoCalc still fails with

    2023-11-21T19:13:42.235Z: cocalc:debug:email send_email(to:te...@organization.com) -- \
    error sending email -- {"code":"ESOCKET","command":"CONN"}

[Harald Schilly]

Hello Matthew,

Unfortunately, I do not have any idea what's really going on. A stackoverflow comment

https://stackoverflow.com/questions/56147491/what-does-esocket-error-mean-when-im-trying-to-send-an-email

pointed here

https://github.com/nodemailer/nodemailer/issues/889#issuecomment-488267379

Using the IP address instead of a host, tls certificates, or something with a firewall blocking connections, is mentioned.

In general, either this is fundamentally incompatible with nodemailer or there aren't enough configuration options. Without knowing more, we can't help.

My first and only thought is, we need a working configuration for nodemailer, to know this works at all and then we compare what's different from CoCalc's configuration. Just a simple, few lines long, javascript code setting up a nodemailer transporter and sending a test email would help (please redact the credentials, though :-) )

-- Harald

You received this message because you are subscribed to the Google Groups "CoCalc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cocalc+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cocalc/794bcb63-1f8f-4ec5-ac24-7194d4b51558n%40googlegroups.com.

[Matthew Saltzman]

I haven't played with Nodemailer yet, but I have some info to add from the Postfix interaction side. When I connect via port 25 or 587 (with secure set to no in CoCalc), systemctl status reports:

Nov 25 18:38:09 erdospal.math.clemson.edu postfix/master[8059]: daemon started -- version 3.8.1, configuration /etc/postfix
Nov 25 18:38:09 erdospal.math.clemson.edu systemd[1]: Started postfix.service - Postfix Mail Transport Agent.
Nov 25 18:40:22 erdospal.math.clemson.edu postfix/submission/smtpd[8362]: connect from unknown[172.17.0.2]
Nov 25 18:40:22 erdospal.math.clemson.edu postfix/submission/smtpd[8362]: SSL_accept error from unknown[172.17.0.2]: lost connection
Nov 25 18:40:22 erdospal.math.clemson.edu postfix/submission/smtpd[8362]: lost connection after STARTTLS from unknown[172.17.0.2]
Nov 25 18:40:22 erdospal.math.clemson.edu postfix/submission/smtpd[8362]: disconnect from unknown[172.17.0.2] ehlo=1 starttls=0/1 commands=1/2
Nov 25 18:41:42 erdospal.math.clemson.edu postfix/smtps/smtpd[8377]: connect from unknown[172.17.0.2]
Nov 25 18:41:42 erdospal.math.clemson.edu postfix/smtps/smtpd[8377]: SSL_accept error from unknown[172.17.0.2]: lost connection
Nov 25 18:41:42 erdospal.math.clemson.edu postfix/smtps/smtpd[8377]: lost connection after CONNECT from unknown[172.17.0.2]
Nov 25 18:41:42 erdospal.math.clemson.edu postfix/smtps/smtpd[8377]: disconnect from unknown[172.17.0.2] commands=0/0

When I connect to port 465 (secure = yes), the TLS connection is presumably active, and the drop occurs right after the connection.

[Matthew Saltzman]

Got it working! Here are some hints, in case anyone stumbles on this thread looking for them.

• The problem was resolved without changing anything in Nodemailer. Issues were all on the Postfix side.
  
• CoCalc wants a TLS connection to the SMTP server. TLS requires that the server have a valid certificate--self-generated ones won't do. Look at Let's Encrypt if you don't have one but can expose a web server on port 80 or you have control of your DNS configuration. (I didn't have either, but I get certs from my enterprise.)
  
• When you configure the server in CoCalc, you must use the DNS name that corresponds to the certificate your server uses.
• Error messages from both CoCalc and Postfix are singularly unhelpful. Debug TLS connection issues with  "gnutls-cli --starttls-proto smtp --port 587 server.name".
• CoCalc seems to insist on authenticating, even if you don't provide a userid and password in CoCalc. (This is what led me to try to configure my own SMTP server in the first place. My enterprise smart host doesn't do AUTH.) The server side needs to be configured accordingly.
  
• If you're debugging from the command line, the string to pass to AUTH PLAIN can be formed with "echo -ne "\0username\0password"|base64", which produces a string like "AHVzZXJuYW1lAHBhc3N3b3Jk".
• I ended up configuring port 587 with optional TLS encryption. Then in CoCalc, I use port 587 with security off. I think that causes CoCalc to invoke STARTTLS on the session explicitly. Port 25 in Postfix doesn't do AUTH, at least by default. Port 465 is for SMTPS connections.

If any of that is wrong, feel free to correct me, please. 

Thanks for the help here. I did get useful ideas for what to look for.