Elasticsearch 7.13 for Mailroom 6.2.3 error. Maybe related to API changes

[Ubuhinga Vizion]

Hi everyone,

I was running Rapidpro smoothly but yesterday after an upgrade of the system, Mailroom started producing some errors. 

Jun  5 14:03:52 rapidpro mailroom[21161]: time="2021-06-05T14:03:52+02:00" level=error msg="error handling request" error="no elastic client available, check your configuration" http_request="&{POST /mr/contact/search HTTP/1.1 1 1 map[Accept:[*/*] Accept-Encoding:[gzip, deflate] Authorization:[Token XXXXXXXXXX] Connection:[keep-alive] Content-Length:[133] Content-Type:[application/json] User-Agent:[Temba]] 0xc0002de300 <nil> 133 [] false localhost:8090 map[] map[] <nil> map[] 127.0.0.1:48336 /mr/contact/search <nil> <nil> <nil> 0xc00033e180}"

Looking further, I saw that Rapidpro 6.2.4 has support for ElasticSearch 7. And I can see that I can access the Elastic endpoints. 

curl -X GET 'http://localhost:9200'

{

  "name" : "elasticsearch-ubuntu20-04",

  "cluster_name" : "elasticsearch",

  "cluster_uuid" : "iuJkaAubTDG6zJXh3GFGgQ",

  "version" : {

    "number" : "7.13.1",

    "build_flavor" : "default",

    "build_type" : "deb",

    "build_hash" : "9a7758028e4ea59bcab41c12004603c5a7dd84a9",

    "build_date" : "2021-05-28T17:40:59.346932922Z",

    "build_snapshot" : false,

    "lucene_version" : "8.8.2",

    "minimum_wire_compatibility_version" : "6.8.0",

    "minimum_index_compatibility_version" : "6.0.0-beta1"

  },

  "tagline" : "You Know, for Search"

}

So I'm wondering if my error is not related to these changes https://github.com/elastic/elasticsearch/issues/69532 .

To solve that, do I need to downgrade Elasticsearch ?  

Thanks.

[Nic Pottier]

We're using 7.10.* so it is possible the wire protocol is somehow different, but I'd first check that you are passing in the right host etc, to mailroom when starting it.

Try starting mailroom with `-debug-conf` that will show you the settings it is using and you can double check your Elastic settings match.

--
You received this message because you are subscribed to the Google Groups "rapidpro-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rapidpro-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rapidpro-dev/4699186c-d1c7-4375-a1ef-c278b072e1afn%40googlegroups.com.

[Ubuhinga Vizion]

Well. We have a lot of errors. Mailroom sometimes hangs and doesn't finish a flow.

 Again, all the system was running well a week ago. I didn't know which version of ElasticSearch we were running but definitely 7.10+. And the strangest thing is sometimes the link goes through. I mean, for example when typing a contact in the omnibox, sometimes it works, other times it doesn't. And the error shown is always either 

"500 Server Error: Internal Server Error for url: http://localhost:8090/mr/contact/search"

"500 Server Error: Internal Server Error for url: http://localhost:8090/mr/sim/start"

....

But our environment variables are Ok.

in /etc/environment

MAILROOM_AUTH_TOKEN=XXXXXXXXXXXXX

MAILROOM_DB="postgres://temba:somepa...@db-postgresql.ondigitalocean.com:25067/rapid5pool?sslmode=require"

MAILROOM_SMTP_SERVER=smtp://domain%40somep...@smtp.gmail.com:587/?from=domain%40gmail.com

MAILROOM_LOG_LEVEL=error

MAILROOM_S3_REGION=eu-central-1

MAILROOM_S3_MEDIA_BUCKET=uczmzufdijvhywvo

MAILROOM_S3_MEDIA_PREFIX=attachments

MAILROOM_AWS_ACCESS_KEY_ID=AWSIDACCESSS

MAILROOM_AWS_SECRET_ACCESS_KEY="AWSSECRET"

MAILROOM_ADDRESS=localhost

MAILROOM_DOMAIN=subdomain.domain.com

MAILROOM_ATTACHMENT_DOMAIN=subdomain.domain.com

MAILROOM_ELASTIC="http://localhost:9200"

MAILROOM_FCM_KEY="FirebaseCloudMessagingKey"

in settings.py

MAILROOM_URL = os.environ.get('MAILROOM_URL', 'http://localhost:8090')

MAILROOM_AUTH_TOKEN = "XXXXXXXXXXXXX"

And finally, in nginx

upstream rapidpro_server {

  server unix:/var/www/envs/rapidpro/rapidpro.sock;

}

upstream courier_server {

  server 127.0.0.1:8080 fail_timeout=60;

}

upstream mailroom_server {

  server 127.0.0.1:8090 fail_timeout=60;

}

server {

    listen 80 default_server;

    listen [::]:80 default_server ipv6only=on;

    server_name xxx.xxx.xxx.xxx subdomain.domain.com ;

    client_max_body_size 500M;

    location = /favicon.ico { access_log off; log_not_found off; }

    access_log /var/log/nginx/rapidpro.access.log;

    error_log /var/log/nginx/rapidpro.error.log;

    location ~ /.well-known {

                allow all;

    }

    location /robots.txt {

        return 200 "User-agent: *\nDisallow: /";

    }

    location /media {

        autoindex on;

        alias /var/www/prod/rapidpro/media;

        }

    location /sitestatic {

        autoindex on;

        alias /var/www/prod/rapidpro/sitestatic;

    }

    # all Mailroom URLs go to Mailroom

    location ^~ /mr/ {

        proxy_set_header Host $http_host;

        proxy_pass http://mailroom_server;

        break;

    }

    # all courier URLs go to courier

    location ^~ /c/ {

        proxy_set_header Host $http_host;

        proxy_pass http://courier_server;

        break;

    }

    location / {

        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_set_header Host $http_host;

        proxy_redirect off;

        if (!-f $request_filename) {

            proxy_pass http://rapidpro_server;

            break;

        }

   }

    listen [::]:443 ssl ipv6only=on; # managed by Certbot

    listen 443 ssl; # managed by Certbot

    ssl_certificate /etc/letsencrypt/live/subdomain.domain.com/fullchain.pem; # managed by Certbot

    ssl_certificate_key /etc/letsencrypt/live/subdomain.domain.com/privkey.pem; # managed by Certbot

    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot

    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

ElasticSearch is well responding as well. 

Previously I had managed to solve the error by running the command

./rp-indexer   -rebuild

But now that doesn't help at all. 

[Ubuhinga Vizion]

Maybe it's this API changes from 7.10 to 7.11 that are breaking some the Mailrom services.