Error "getaddrinfo ENOTFOUND" during setup, when installing from source

[Eemeli Aro]

Hi!

I'm trying to install Sandstorm from source on an Ubuntu 16.04 machine running on Azure, but I'm getting the following errors in the Sandstorm system log when trying to complete the setup (built from v0.196, commit 82498f6):

When trying to fetch "pre-installed" apps:

Exception in setTimeout callback: Error: getaddrinfo ENOTFOUND app-index.sandstorm.io app-index.sandstorm.io:443
    at Object.Future.wait (/programs/server/node_modules/fibers/future.js:449:15)
    at Object.call (packages/meteor/helpers.js:119:1)
    at Object.HTTP.get (packages/http/httpcall_common.js:50:20)
    at SandstormDb.updateAppIndex (packages/sandstorm-db/db.js:1717:23)
    at startPreinstallingAppsHelper (imports/server/migrations.js:673:8)
    at [object Object]._.extend.withValue (packages/meteor/dynamics_nodejs.js:56:1)
    at packages/meteor/timers.js:6:1
    at runWithEnvironment (packages/meteor/dynamics_nodejs.js:110:1)
    - - - - -
    at errnoException (dns.js:27:10)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:78:26)

And when trying to log in using Google or Github:

{"line":"398","file":"oauth_server.js","message":"Error in OAuth Server: Failed to complete OAuth handshake with Google. getaddrinfo ENOTFOUND accounts.google.com accounts.google.com:443","time":{"$date":1479813890728},"level":"warn"}
Exception while invoking method 'login' Error: Failed to complete OAuth handshake with Google. getaddrinfo ENOTFOUND accounts.google.com accounts.google.com:443
    at getTokens (packages/google/google_server.js:59:1)
    at Object.handleOauthRequest (packages/google/google_server.js:10:1)
    at OAuth._requestHandlers.(anonymous function) (packages/oauth2/oauth2_server.js:8:1)
    at middleware (packages/oauth/oauth_server.js:173:1)
    at packages/oauth/oauth_server.js:146:1

My sandstorm.conf file is as follows:

SERVER_USER=eemeli
PORT=6080
MONGO_PORT=6081
BIND_IP=0.0.0.0
BASE_URL=http://custom-url-here.fi:6080
WILDCARD_HOST=*.custom-url-here.fi:6080
UPDATE_CHANNEL=none
ALLOW_DEV_ACCOUNTS=false
SMTP_LISTEN_PORT=30025

The server has inbound ports 80, 443, and 6080 open, and curl'ing from there to any of the apparently not-found domains (e.g. https://accounts,google.com, https://github.com) is successful. Getting an admin token and signing in with it do succeed at the custom URL, and configuring Google & Github as account providers appears to persist across Sandstorm service restarts.

Is there something I perhaps missed in the setup, or something additional info that might help with figuring out what's happening here? I've tried running the test suite, but am currently stuck with that at Selenium timing out its connection to Firefox (v50.0), which I'll next try to downgrade to 47.

Btw, it'd be nice if the dependencies for the tests were listed somewhere, rather than needing to discover them one by one. Also, at least the default Ubuntu vagrant version does not appear to be compatible with the libvirt plugin (for the installer tests), but the upstream version works fine.

eemeli

[Eemeli Aro]

For a bit more info, I got the tests to run (using Firefox 45, which was the alternative available via apt-get), but they fail pretty uniformly with the same "getaddrinfo ENOTFOUND" error causing the <#step-confirm> element test to time out.

On the same system, Sandstorm installed via

curl https://install.sandstorm.io | bash

works as expected (with the same config as above), and does not experience any problems getting a network connection.

--
You received this message because you are subscribed to the Google Groups "Sandstorm Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sandstorm-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sandstorm-dev/38432709-8ea6-4e77-9cf7-fea4013de0f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kenton Varda]

Hi Eemeli,

Hmm, it sounds to me like for some reason Sandstorm is unable to make DNS queries. It sounds like DNS works on the machine in general, so probably the problem specifically occurs inside Sandstorm's self-container, in which the whole Sandstorm server runs.

One possibility is that your server uses an usual DNS mechanism -- one that doesn't involve querying a standard DNS server.

Can you tell us the contents of:

- /etc/resolv.conf

- /etc/nsswitch.conf (specifically the "hosts:" line)

This will tell us if your DNS is configured "interestingly".

-Kenton

To view this discussion on the web visit https://groups.google.com/d/msgid/sandstorm-dev/CAFuwTa7YRRcRzdS-uFcMdEd09CoevB7KhoBTeDb36HKhdk0yAA%40mail.gmail.com.

[Eemeli Aro]

On 27 November 2016 at 01:42, Kenton Varda <ken...@sandstorm.io> wrote:

Hi Eemeli,

Hmm, it sounds to me like for some reason Sandstorm is unable to make DNS queries. It sounds like DNS works on the machine in general, so probably the problem specifically occurs inside Sandstorm's self-container, in which the whole Sandstorm server runs.

It's also a bit odd that the error only shows up if I try to compile Sandstorm from source myself; running the normal install of Sandstorm runs without any network problems.

One possibility is that your server uses an usual DNS mechanism -- one that doesn't involve querying a standard DNS server.

Can you tell us the contents of:

- /etc/resolv.conf

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)

#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN

nameserver 127.0.0.1

search 3eavhvw2ygfejjnlsaeuibokjd.fx.internal.cloudapp.net

 

- /etc/nsswitch.conf (specifically the "hosts:" line)

# /etc/nsswitch.conf

#

# Example configuration of GNU Name Service Switch functionality.

# If you have the `glibc-doc-reference' and `info' packages installed, try:

# `info libc "Name Service Switch"' for information about this file.

passwd:         compat

group:          compat

shadow:         compat

gshadow:        files

hosts:          files dns

networks:       files

protocols:      db files

services:       db files

ethers:         db files

rpc:            db files

netgroup:       nis

 

This will tell us if your DNS is configured "interestingly".

Thank you for helping out on this!

eemeli

[Kenton Varda]

On Sun, Nov 27, 2016 at 3:45 AM, Eemeli Aro <eemel...@vincit.fi> wrote:

On 27 November 2016 at 01:42, Kenton Varda <ken...@sandstorm.io> wrote:

Hi Eemeli,

Hmm, it sounds to me like for some reason Sandstorm is unable to make DNS queries. It sounds like DNS works on the machine in general, so probably the problem specifically occurs inside Sandstorm's self-container, in which the whole Sandstorm server runs.

It's also a bit odd that the error only shows up if I try to compile Sandstorm from source myself; running the normal install of Sandstorm runs without any network problems.

Oh, that's interesting! I missed that in your original message. OK, so it seems like make-bundle.sh is failing to detect some DNS dependency when building on your system. It does some weird tricks here so it doesn't totally surprise me that they'd go wrong, e.g. it tries to do a DNS lookup while stracing a binary to see what libraries it opens.

Can you run:

    find /opt/sandstorm/latest/ > sandstorm-files.txt

And then gzip the output and send it to me / attach it to a reply to this thread?

Out of curiosity, why did you choose to build from source?

-Kenton

[Eemeli Aro]

Sure; it's attached. diff'ing that to the file list of a normally-installed sandstorm on the same machine, I see at least some version differences and a few missing files in lib/.

Out of curiosity, why did you choose to build from source?

We're building a custom web tool for a client, mainly around Wekan to start with, and are using Sandstorm effectively as a security wrapper around it. It's looking like we'll want to customise at least the shell client's appearance, but may end up wanting to do other stuff as well. Nothing so far has actually required us to do any system-level changes, but I want to make sure we can, if needed. Also, I'm just more comfortable using a system that I can verifiably build for myself. :)

-Kenton

[Eemeli Aro]

On 7 December 2016 at 14:09, Eemeli Aro <eemel...@vincit.fi> wrote:

Oh, that's interesting! I missed that in your original message. OK, so it seems like make-bundle.sh is failing to detect some DNS dependency when building on your system. It does some weird tricks here so it doesn't totally surprise me that they'd go wrong, e.g. it tries to do a DNS lookup while stracing a binary to see what libraries it opens.

I managed to solve this on my own; the missing dependency was libnss_dns.so.2, which does exist on the host system (Ubuntu 16.04.1, under /lib/x86_64-linux-gnu) but is not included in the (successful) dnstest strace output that's used by make-bundle.sh to map the getaddrinfo dependencies. That file does show up when running strace tmp/dnstest on at least Ubuntu 14.04 and RHEL 7.3, the other Linux systems I had easy access to.

So, adding the following hack-y line to make-bundle.sh got me a working system:

echo '/lib/x86_64-linux-gnu/libnss_dns.so.2' | copyDeps

[Kenton Varda]

Weird, how does the test program get away without loading libnss_dns? Maybe you could send me a copy of that strace output?

-Kenton

--

You received this message because you are subscribed to the Google Groups "Sandstorm Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sandstorm-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sandstorm-dev/CAFuwTa7TB_J-QM2MDnzoNktqsN4g2_yiVn4Vwpf0jj%3DpdhkuEw%40mail.gmail.com.

[Kenton Varda]

Turns out if nscd is running locally then glibc never loads libnss. See:

https://github.com/sandstorm-io/sandstorm/issues/2814

-Kenton