RE: [fixmystreet] Digest for fixmystreet@mysociety.org - 1 update in 1 topic

[Richard Perry]

Thanks Matthew,

 

The AWS AMI (ID 0baac5517e963f70a) works very well, installed and added key pair, allocated elastic IP and opened ports and good to go.

Very quick and available painlessly.

 

Docs sort of indicate email should be available, but appears not.

No problem, sure we can configure that.

Now into config steps and setting up superadmin using scripts in /bin/

 

Not tried docker yet, but will also try this.

 

Richard

 

 

From: fixmy...@mysociety.org <fixmy...@mysociety.org>
Sent: Wednesday, November 20, 2024 2:08 AM
To: Digest recipients <fixmy...@mysociety.org>
Subject: [fixmystreet] Digest for fixmy...@mysociety.org - 1 update in 1 topic

 

fixmy...@mysociety.org

Google Groups

Topic digest
View all topics

·        Version 6.0 released - 1 Update

Version 6.0 released

Matthew Somerville <mat...@mysociety.org>: Nov 19 04:38PM Hi,   We've now got a new AMI uploaded (ID 0baac5517e963f70a), and have updated the install documentation to point directly to it, hope that's helpful.   I've updated the broken link on hub.docker.com, thanks. There's nothing to change in the Docker documentation, as far as I'm aware, there's nothing version specific in them.   ATB, Matthew

Back to top

You received this digest because you're subscribed to updates for this group. You can change your settings on the group membership page. To unsubscribe from this group and stop receiving emails from it send an email to fixmystreet...@mysociety.org.

 

Virus-free.www.avg.com

[Richard Perry]

Unfortunately, hit problems with the configuration and creation of superadmin.

Following directions here:

https://fixmystreet.org/install/ami/

The issue is a permissions issue with postgres database access running:

~/fixmystreet$ bin/createsuperuser fmsa...@example.org password

The issue we found is the fms database does in the general.yml does (FMS_DB_PASS: ) does not have a password value. So, we can connect using SSH to the app, but he app can not access the database.

Trying to follow these instructions on manual install then also fail:

https://fixmystreet.org/install/database/

Access:

/etc/postgresql/13/main/pg_hba.conf

Set user

Local fms fms md5

restart

$ sudo service postgresql restart

set up cmd line access:

~/.pgpass

localhost:*:fms:fms:somepassword

Password is same as general.yml, but this value is blank then this also needs to be blank, but wont work.

So, next step, try and set a new password via CMD line for db user using an appropriately edited version of:

ALTER USER fms WITH PASSWORD 'pword';

Complete the verification steps to check access to database, but createsuperuser script still fails.

==

So, in summary:

1. Cant create a superadmin as the script fails
2. Email service is not working so cant verify any standard users.

Hoping you can help please,

Richard

[Matthew Somerville]

On Fri, 22 Nov 2024 at 18:16, 'Richard Perry' wrote:

Following directions here: https://fixmystreet.org/install/ami/

The issue is a permissions issue with postgres database access running:

~/fixmystreet$ bin/createsuperuser fmsa...@example.org password

You don't provide the error message when running this, or full command line to know which user you were running this as.

I have just spun up a new AMI and it appears to work fine, here's everything I typed:

-------

$ ssh -i ~/Downloads/Amazon\ 2024-11-22.pem admin@[snip]
admin@ip-172-31-28-181:~$ sudo su - fms
fms@ip-172-31-28-181:~$ cd fixmystreet
fms@ip-172-31-28-181:~/fixmystreet$ bin/createsuperuser fmsa...@example.org password
fmsa...@example.org is now a superuser.
fms@ip-172-31-28-181:~/fixmystreet$
-------

If I try as the 'admin' user I get a [Peer authentication failed for user "fms"] error. So I'm not sure what you did if that doesn't work for you.

[snip - you shouldn't have to mess around with postgresql permissions]

As for email - I don't know what you mean by "not working", but what the AMI does is set up postfix listening on port 25; if you haven't allowed outgoing connections, the mail can't then be sent anywhere (and I doubt a random spun up EC2 should be sending email anyway, I imagine they're heavily black listed due to spam etc). You can look in /var/log/mail.log to see what it's doing, anyway. If you want to use a different SMTP server, which you'll have to for an EC2 server, for example the SES SMTP server, then you'll need to edit your general.yml configuration to the details of whatever SMTP server you wish to use, and then restart the service, and hopefully that would then work as you wish.

ATB,

Matthew

[Richard Perry]

Thanks Matthew,

No idea what the issue was with the first attempt with the AMI.

After your message we created a new server using the AMI and IP, A record and security group settings for the first EC2 instance and could SSH in and create the superuser straight away.

As you say, painless....

thanks,

Richard

[Richard Perry]

We have made progress, but seeing issues with setting up the following:

1 - EMAIL using SES
We have set up AWS SES with a verified domain and emails (to and from)
The SMTP details are verified as ok using AWS SES test as well as a third-party test service
We are using these verified emails (as in sandbox) for new user creation as well as new issue raising.
We are not receiving emails to the SES verified email accounts. We are also not receiving using postfix.
Link to is an obfuscated copy of general.yml (API keys are correct in our version, but removed, and we use xxx in some values to obfuscate).

We are using a cobrand of "fixmebcp" and have set up the child theme in:
/var/www/fixmystreet/fixmystreet/templates/web/fixmebcp
as well as templates at:
/var/www/fixmystreet/fixmystreet/templates/email/fixmebcp

Questions:
There are a number of send methods in the dropdown, and we should be "email" and not one of the other email options, is that correct?


What else are we missing please, as pulling our hair out?

thanks,

Richard

[Richard Perry]

Sorry, forgot to add latest mail log entries:

Dec 20 17:00:54 ip-172-31-41-2 postfix/smtp[19523]: warning: SASL authentication failure: No worthy mechs found
Dec 20 17:00:54 ip-172-31-41-2 postfix/smtp[19523]: A2FDD41020: SASL authentication failed; cannot authenticate to server email-smtp.eu-west-1.amazonaws.com[34.240.193.206]: no mechanism available
Dec 20 17:00:54 ip-172-31-41-2 postfix/smtp[19515]: warning: SASL authentication failure: No worthy mechs found
Dec 20 17:00:54 ip-172-31-41-2 postfix/smtp[19515]: 9D23B41028: SASL authentication failed; cannot authenticate to server email-smtp.eu-west-1.amazonaws.com[52.18.165.220]: no mechanism available
Dec 20 17:00:54 ip-172-31-41-2 postfix/smtp[19523]: warning: SASL authentication failure: No worthy mechs found
Dec 20 17:00:54 ip-172-31-41-2 postfix/smtp[19523]: A2FDD41020: to=<f...@qpc.org.uk>, orig_to=<fms>, relay=email-smtp.eu-west-1.amazonaws.com[99.81.178.80]:587, delay=353, delays=352/0.03/0.35/0, dsn=4.7.0, status=deferred (SASL authentication failed; cannot authenticate to server email-smtp.eu-west-1.amazonaws.com[99.81.178.80]: no mechanism available)
Dec 20 17:00:54 ip-172-31-41-2 postfix/smtp[19515]: warning: SASL authentication failure: No worthy mechs found
Dec 20 17:00:54 ip-172-31-41-2 postfix/smtp[19515]: 9D23B41028: to=<f...@qpc.org.uk>, orig_to=<fms>, relay=email-smtp.eu-west-1.amazonaws.com[99.81.178.80]:587, delay=533, delays=532/0/0.42/0, dsn=4.7.0, status=deferred (SASL authentication failed; cannot authenticate to server email-smtp.eu-west-1.amazonaws.com[99.81.178.80]: no mechanism available)
Dec 20 17:02:01 ip-172-31-41-2 postfix/pickup[18338]: ABD1B41030: uid=1001 from=<fms>
Dec 20 17:02:01 ip-172-31-41-2 postfix/cleanup[19538]: ABD1B41030: message-id=<202412201702...@qpc.org.uk>
Dec 20 17:02:01 ip-172-31-41-2 postfix/qmgr[18339]: ABD1B41030: from=<f...@qpc.org.uk>, size=782, nrcpt=1 (queue active)
Dec 20 17:02:01 ip-172-31-41-2 postfix/error[19539]: ABD1B41030: to=<f...@qpc.org.uk>, orig_to=<fms>, relay=none, delay=0.02, delays=0.02/0/0/0, dsn=4.7.0, status=deferred (delivery temporarily suspended: SASL authentication failed; cannot authenticate to server email-smtp.eu-west-1.amazonaws.com[52.18.165.220]: no mechanism available)
Dec 20 17:05:01 ip-172-31-41-2 postfix/pickup[18338]: B188941032: uid=1001 from=<fms>
Dec 20 17:05:01 ip-172-31-41-2 postfix/cleanup[19560]: B188941032: message-id=<202412201705...@qpc.org.uk>
Dec 20 17:05:01 ip-172-31-41-2 postfix/qmgr[18339]: B188941032: from=<f...@qpc.org.uk>, size=784, nrcpt=1 (queue active)
Dec 20 17:05:01 ip-172-31-41-2 postfix/error[19562]: B188941032: to=<f...@qpc.org.uk>, orig_to=<fms>, relay=none, delay=0.03, delays=0.02/0/0/0.01, dsn=4.7.0, status=deferred (delivery temporarily suspended: SASL authentication failed; cannot authenticate to server email-smtp.eu-west-1.amazonaws.com[52.18.165.220]: no mechanism available)
Dec 20 17:05:53 ip-172-31-41-2 postfix/qmgr[18339]: 98B4A41026: from=<f...@qpc.org.uk>, size=784, nrcpt=1 (queue active)
Dec 20 17:05:53 ip-172-31-41-2 postfix/qmgr[18339]: 935FE41022: from=<f...@qpc.org.uk>, size=818, nrcpt=1 (queue active)
Dec 20 17:05:53 ip-172-31-41-2 postfix/qmgr[18339]: A78AA4102E: from=<f...@qpc.org.uk>, size=784, nrcpt=1 (queue active)
Dec 20 17:05:53 ip-172-31-41-2 postfix/qmgr[18339]: 7F21E4101A: from=<f...@qpc.org.uk>, size=784, nrcpt=1 (queue active)
Dec 20 17:05:53 ip-172-31-41-2 postfix/error[19562]: 98B4A41026: to=<f...@qpc.org.uk>, orig_to=<fms>, relay=none, delay=952, delays=952/0/0/0, dsn=4.7.0, status=deferred (delivery temporarily suspended: SASL authentication failed; cannot authenticate to server email-smtp.eu-west-1.amazonaws.com[52.18.165.220]: no mechanism available)
Dec 20 17:05:53 ip-172-31-41-2 postfix/error[19562]: 935FE41022: to=<f...@qpc.org.uk>, orig_to=<fms>, relay=none, delay=1132, delays=1132/0.01/0/0, dsn=4.7.0, status=deferred (delivery temporarily suspended: SASL authentication failed; cannot authenticate to server email-smtp.eu-west-1.amazonaws.com[52.18.165.220]: no mechanism available)
Dec 20 17:05:53 ip-172-31-41-2 postfix/error[19569]: A78AA4102E: to=<f...@qpc.org.uk>, orig_to=<fms>, relay=none, delay=352, delays=352/0.01/0/0, dsn=4.7.0, status=deferred (delivery temporarily suspended: SASL authentication failed; cannot authenticate to server email-smtp.eu-west-1.amazonaws.com[52.18.165.220]: no mechanism available)
Dec 20 17:05:53 ip-172-31-41-2 postfix/error[19562]: 7F21E4101A: to=<f...@qpc.org.uk>, orig_to=<fms>, relay=none, delay=2152, delays=2152/0.01/0/0.01, dsn=4.7.0, status=deferred (delivery temporarily suspended: SASL authentication failed; cannot authenticate to server email-smtp.eu-west-1.amazonaws.com[52.18.165.220]: no mechanism available)

Search on google appears to indicate an issue with SASL authentication in postfix.

Tried following these steps, but still cant resolve:

https://stackoverflow.com/questions/26447316/mac-os-x-10-10-yosemite-postfix-sasl-authentication-failed

Best regards,

Richard

[Matthew Somerville]

Hi,

There appear to be two different things here, I'm not sure which one you're wanting to go with.

In the one case, if you had FixMyStreet SMTP settings set to send to "localhost" (which you don't in what you posted), and then postfix is failing to relay email to SES, then that points to an issue with your postfix setup. I haven't used Amazon SES myself, but it appears to have detailed instructions on how to set it up with postfix here, which I'm not sure if you found or not:

    https://docs.aws.amazon.com/ses/latest/dg/postfix.html

So perhaps following that will let you get email via postfix working, with FixMyStreet then sending to localhost, and that's one way of it working.

In the other case, with the configuration you posted it's set up to send email directly from FixMyStreet to SES, so then postfix isn't involved (unless I've misunderstood something). You haven't posted any errors or anything from that approach - is there anything in your development server output when you try this way, or in any web server error logs? If there's an error sending email directly, I would assume it would log this somewhere. I couldn't say what might not be working with this method without information as to what issue you are having. But hopefully you get the postfix way working and then this doesn't matter.

The default send method is "Email", so you don't need to select anything from the drop down, but Email would be the right one to pick, yes.

ATB,

Matthew

On Fri, 20 Dec 2024 at 17:22, Richard Perry <rich...@lumispec.eu> wrote:

Sorry, forgot to add latest mail log entries:

Dec 20 17:00:54 ip-172-31-41-2 postfix/smtp[19523]: warning: SASL authentication failure: No worthy mechs found
Dec 20 17:00:54 ip-172-31-41-2 postfix/smtp[19523]: A2FDD41020: SASL authentication failed; cannot authenticate to server email-smtp.eu-west-1.amazonaws.com[34.240.193.206]: no mechanism available

[snip] 

[Richard Perry]

Hi Matthew,

Thanks for the fast response.

We installed AWS SES as postfix was not sending emails after installing the AMI.

At this stage it is unimportand if we use postfix on local or SES, we just cant create users or issues as emails are not being sent.

I incorrectly assumed that SES values in general.yml would set the SES service up with authentication etc. Appears this is not the case, and both postix and SES need to be set up via CLI - is that correct?

The AWS link sent requires same info for URL, port and authentication.

If we need to use CLI, then is there a reason the following is also in general.yml?
Manual setup of SES in postfix via CLI does the same thing (but is probably more secure as step 6 in AWS steps is securing the SMTP password and security token that would otherwise be visible in the general.yml file), so why in 2 places?

SMTP_SMARTHOST: 'email-smtp.eu-west-1.amazonaws.com'

SMTP_TYPE: 'tls'

SMTP_PORT: '587'  # You can use 587 for STARTTLS

SMTP_USERNAME: 'WeAreUsingIAMUsernameProvided'

SMTP_PASSWORD: 'WeAreUsingIAMPasswordProvided'

So, if we use CLI set up reinstall and then configure postfix, then configure postfix to use SES (as per AWS link) then we just leave these settings in general.yml blank, is that correct?

Best regards,
Richard 

[Matthew Somerville]

Hi,

"At this stage it is unimportant if we use postfix on local" - as you're using EC2 I imagine you will have to send email via some external mail gateway (whether that's SES or whatever); you can't send email directly from an EC2 server with any hope that email will be delivered, because EC2 servers are too easily used for spam.

I'm sorry if I wasn't clear enough in my previous email. FixMyStreet sends email according to what you provide in its configuration:

* If you provide SES SMTP details/credentials in the FixMyStreet configuration, FixMyStreet will try and send email directly to that server. Postfix on your server is not involved in this arrangement in that case. If emails aren't being sent in that set up, you haven't provided any details to say what could be happening there, but as far as I'm aware that set up should work assuming your host allows external connections on the correct ports and so on; you'd have to debug that further if that's how you wanted to use it.

* If you provide "localhost" SMTP credentials, then FIxMyStreet will connect to whatever SMTP server is running on the same host, which in your case is postfix. It is then entirely up to postfix and how it is configured what it does with that email.

* If you don't provide any SMTP credentials, then FIxMyStreet tries to send the email to the sendmail binary (which will probably send it to postfix but I'm not sure on that point).

So nothing is or should be in two places; you configure your SES SMTP in one place, wherever you decide you want that to be, either directly in FixMyStreet, or in postfix - the two options from my previous email. It sounds like if you set up postfix to send email to SES and get that working, then you tell FixMyStreet to send email via SMTP to localhost, which postfix will be listening on. This is the default set up of the AMI without any changes, with FixMyStreet sending email to localhost and relying on the mail server being set up however the administrator wants to send email.

ATB,
Matthew

[Richard Perry]

The emails issues we experienced were fixed by doing the following:

1. Set up postfix and add SASL configuration
   as per:  https://docs.aws.amazon.com/ses/latest/dg/postfix.html
   test to make sure that postfix can send emails to external mailboxes.
2. Set up SES configuration in main.cf
3. Configure SES settings in general.yml file
4. Add email templates for cobrand into /templates/email
   FMX should then be able to send emails to external mailboxes