ELSA broken after soup upgrade - mysql access denied

266 views
Skip to first unread message

Kevin Branch

unread,
Jun 23, 2015, 8:43:56 PM6/23/15
to securit...@googlegroups.com
After running soup on a standalone SO system today and rebooting, ELSA reports "0.0 logs indexed and 0.0 archived" and every attempt to use it generates this error in /nsm/elsa/data/elsa/log/web.log:

* ERROR [2015/06/23 22:12:24] /opt/elsa/web/lib/View/API.pm (79) View::API::__ANON__ 9451 [undef]
cannot connect to dbi:mysql:database=syslog;port=3306: Access denied for user 'www-data'@'localhost' (using password: NO) at /opt/elsa/web/lib/SyncMysql.pm line 18.
 at /usr/share/perl5/Ouch.pm line 28
        Ouch::ouch(502, 'cannot connect to dbi:mysql:database=syslog;port=3306: Access...', 'HASH(0x7fc59e3e6198)') called at /usr/share/perl5/Ouch.pm line 33
        Ouch::throw(502, 'cannot connect to dbi:mysql:database=syslog;port=3306: Access...', 'HASH(0x7fc59e3e6198)') called at /opt/elsa/web/lib/Utils.pm line 565
        Utils::_get_db('Controller::Peers=HASH(0x7fc5a129cff8)', 'CODE(0x7fc59e40d0a0)') called at /opt/elsa/web/lib/Utils.pm line 530
        Utils::_get_info('Controller::Peers=HASH(0x7fc5a129cff8)', 1) called at /opt/elsa/web/lib/Controller/Peers.pm line 37
        Controller::Peers::__ANON__() called at /usr/share/perl5/Try/Tiny.pm line 76
        eval {...} called at /usr/share/perl5/Try/Tiny.pm line 67
        Try::Tiny::try('CODE(0x7fc59e3f9e68)', 'Try::Tiny::Catch=REF(0x7fc59e3d8f90)') called at /opt/elsa/web/lib/Controller/Peers.pm line 42
        Controller::Peers::local_info('Controller::Peers=HASH(0x7fc5a129cff8)', 'HASH(0x7fc5a0e8e758)', 'CODE(0x7fc59e3d90b0)') called at /opt/elsa/web/lib/View/API.pm line 121
        View::API::__ANON__() called at /usr/share/perl5/Try/Tiny.pm line 76
        eval {...} called at /usr/share/perl5/Try/Tiny.pm line 67
        Try::Tiny::try('CODE(0x7fc59e3e1fa0)', 'Try::Tiny::Catch=REF(0x7fc5a2c81700)') called at /opt/elsa/web/lib/View/API.pm line 135
        View::API::__ANON__('CODE(0x7fc59e3e1fe8)') called at /usr/share/perl5/Plack/Util.pm line 324
        Plack::Util::__ANON__('CODE(0x7fc5a2b287e8)') called at /usr/share/perl5/Plack/Handler/Apache2.pm line 68
        Plack::Handler::Apache2::call_app('Plack::Handler::Apache2', 'Apache2::RequestRec=SCALAR(0x7fc59cbf7eb8)', 'CODE(0x7fc5a2ddd248)') called at /usr/share/perl5/Plack/Handler/Apache2.pm line 91
        Plack::Handler::Apache2::handler('Apache2::RequestRec=SCALAR(0x7fc59cbf7eb8)') called at -e line 0
        eval {...} called at -e line 0

That is about the only kind of ERROR that pops in in this log.

I totally don't get why ELSA would try to use the www-data account to authenticate with the mysql syslog database.  On another recently soup-upgraded SO box where ELSA is still working fine for me, a manual attempt to use www-data for accessing that database fails as I would expect it to.

# mysql -u www-data syslog
ERROR 1045 (28000): Access denied for user 'www-data'@'localhost' (using password: NO)

I see that my SO system with the now-broken ELSA has proper credentials for the syslog table configured in /etc/elsa_web.conf

  "nodes": {
    "127.0.0.1": {
      "db": "syslog",
      "username": "elsa",
      "password": "biglog",
      "port": 3306,
      "sphinx_port": 9306
    }
  },

Any thought on what broke?

Kevin

Kevin Branch

unread,
Jun 23, 2015, 11:54:04 PM6/23/15
to securit...@googlegroups.com
Copying /etc/elsa_web.conf in from a known working and up-to-date SO system, followed by a "service apache2 restart", completely fixed my problem.

Maybe something in elsa_web.conf didn't get upgraded quite right by soup or maybe I had something a little off in the file myself in a way that didn't break the old ELSA but loused up the config parsing for the new ELSA.  I'm just glad it works now.

Thanks,
Kevin

Doug Burks

unread,
Jul 3, 2015, 6:53:51 AM7/3/15
to securit...@googlegroups.com
Hi Kevin,

Are you able to provide your original elsa_web.conf?

Had you made any modifications to the defaults?

On Tue, Jun 23, 2015 at 11:54 PM, Kevin Branch
> --
> You received this message because you are subscribed to the Google Groups
> "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.



--
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com

Kevin Branch

unread,
Jul 3, 2015, 10:28:42 AM7/3/15
to securit...@googlegroups.com
Doug,

I was able to reproduce this problem this morning by running soup on a very stock SO server I'm running in the cloud.  After running soup for the first time in probably at least a month, my previously responsive ELSA interface on that server throws this error in response to any query attempts:

cannot connect to dbi:mysql:database=syslog;port=3306: Access denied for user 'www-data'@'localhost' (using password: NO) at /opt/elsa/web/lib/SyncMysql.pm line 18.

I compared my elsa_web.conf file before and after the soup upgrade and found it unchanged.  Was it supposed to be changed as part of the ELSA upgrade?

The only non-stock part of my elsa_web.conf file is in the peers section.  Here is the file in its entirety for your inspection:

{
  "apikeys": {
    "elsa": "065c3d51ec0fdbee39a023d38789c2d9"
  },
  "version": {
    "Author": "mcholste",
    "Date": "2013-12-04 12:00:00 -0400 (Wed, 04 Dec 2013)",
    "Rev": "1090",
    "Sphinx": "Sphinx 2.0.7-id64-dev (rel20-r373)"
  },
  "peers": {
    "127.0.0.1": {
      "url": "https://127.0.0.1:3154/",
      "username": "elsa",
      "apikey": "065c3d51ec0fdbee39a023d38789c2d9"
    },
    "CLIENT1": {
      "url": "https://10.81.92.11:3154/",
      "username": "elsa",
      "apikey": "*REDACTED*"
    },
    "CLIENT2": {
      "url": "https://10.81.92.15:3154/",
      "username": "elsa",
      "apikey": "*REDACTED*"
    },
    "CLIENT3": {
      "url": "https://10.81.92.13:3154/",
      "username": "elsa",
      "apikey": "*REDACTED*"
    },
    "CLIENT4": {
      "url": "https://10.81.92.17:3154/",
      "username": "elsa",
      "apikey": "*REDACTED*"
    }
  },
  "admin_email_address": "root@localhost",
  "connectors": {
  },
  "dashboards": {
  },
  "datasources": {
  },
  "transforms": {
    "whois": {
      "known_subnets": {
        "10.0.0.0": {
          "end": "10.255.255.255",
          "org": "MyOrg"
        },
        "192.168.0.0": {
          "end": "192.168.255.255",
          "org": "MyOrg"
        },
        "172.16.0.0": {
          "end": "172.31.255.255",
          "org": "MyOrg"
        }
      },
      "known_orgs": {
        "MyOrg": {
          "name": "MyOrg",
          "org": "MyOrg",
          "descr": "MyOrg",
          "cc": "US",
          "country": "United States",
          "city": "Anytown",
          "state": "Somestate"
        }
      }
    }
  },
  "plugins": {
    "SNORT": "Info::Snort",
    "WINDOWS": "Info::Windows",
    "URL": "Info::Url",
    "BRO_NOTICE": "Info::Bro"
  },
  "info": {
    "snort": {
      "url_templates": [
      ]
    },
    "url": {
      "url_templates": [
      ]
    },
    "windows": {
      "url_templates": [
      ]
    }
  },
  "max_concurrent_archive_queries": 4,
  "schedule_interval": 60,
  "node_info_cache_timeout": 600,
  "email": {
    "display_address": "norepl...@example.com",
    "base_url": "http://elsa/",
    "subject": "ELSA Alert"
  },
  "link_key": "secret",
  "yui": {
    "local": "inc"
  },
  "meta_db": {
    "dsn": "dbi:mysql:database=elsa_web",
    "username": "elsa",
    "password": "biglog"
  },
  "auth": {
    "method": "security_onion"
  },
  "admin_groups": [
    "system",
    "admin"
  ],
  "auth_db": {
    "dsn": "dbi:mysql:database=securityonion_db",
    "username": "root",
    "password": "",
    "auth_statement": "SELECT PASSWORD(password) FROM user_info WHERE username=?",
    "email_statement": "SELECT email FROM user_info WHERE username=?"
  },
  "peer_id_multiplier": 1000000000000,
  "query_timeout": 100000,
  "nodes": {
    "127.0.0.1": {
      "db": "syslog",
      "username": "elsa",
      "password": "biglog",
      "port": 3306,
      "sphinx_port": 9306
    }
  },
  "pcap_url": "https://45.55.186.244/capme",
  "logdir": "/nsm/elsa/data/elsa/log",
  "buffer_dir": "/nsm/elsa/data/elsa/tmp/buffers",
  "debug_level": "DEBUG",
  "default_start_time_offset": 2,
  "livetail": {
    "poll_interval": 5,
    "time_limit": 3600
  }
}



 


Kevin

Doug Burks

unread,
Jul 3, 2015, 10:45:30 AM7/3/15
to securit...@googlegroups.com
On Fri, Jul 3, 2015 at 10:28 AM, Kevin Branch
<ke...@branchnetconsulting.com> wrote:
> I compared my elsa_web.conf file before and after the soup upgrade and found
> it unchanged. Was it supposed to be changed as part of the ELSA upgrade?

Yes, /var/lib/dpkg/info/securityonion-elsa-extras.postinst should've
updated elsa_web.conf.

Please take a look at the section starting with "# Update to ELSA 1205":
https://github.com/Security-Onion-Solutions/securityonion-elsa-extras/blob/master/debian/postinst#L559-L721

For more information, please see:
https://groups.google.com/d/topic/security-onion-testing/OHhNEapIUgE/discussion

Kevin Branch

unread,
Jul 3, 2015, 12:29:46 PM7/3/15
to securit...@googlegroups.com
Doug,

I found the log record of the soup process failing to update /etc/elsa_web.conf.
I was running ELSA rev 1090 when I initiated the soup process.  I redacted the API keys from the log below.  

It it failing here:

It looks like the JSON config in my elsa_web.conf file was parse-able by ELSA rev 1090 but not by the "/usr/bin/securityonion_elsa_register.rb --migrate-web-1205" process.  Is it possible that this process is assuming one and only one peer is defined in the "peers" section of the config?  This server has multiple peers.  

* Backing up /etc/elsa_web.conf to /etc/elsa_web.conf.20150703.
/usr/lib/ruby/1.9.1/json/common.rb:148:in `parse': 746: unexpected token at '{  "apikeys": {    "elsa": "*REDACTED*"  },  "version": {    "Author": "mcholste",    "Date": "2013-12-04 12:00:00 -0400 (Wed, 04 Dec 2013)",    "Rev": "1090",    "Sphinx": "Sphinx 2.0.7-id64-dev (rel20-r373)"  },  "peers": {    "127.0.0.1": {      "url": "https://127.0.0.1:3154/",      "username": "elsa",      "apikey": "*REDACTED*"    },    "CLIENT2": {      "url": "https://10.81.92.15:3154/",      "username": "elsa",      "apikey": "*REDACTED*"    },  },  "admin_email_address": "root@localhost",  "connectors": {  },  "dashboards": {  },  "datasources": {  },  "transforms": {    "whois": {      "known_subnets": {        "10.0.0.0": {          "end": "10.255.255.255",          "org": "MyOrg"        },        "192.168.0.0": {          "end": "192.168.255.255",          "org": "MyOrg"        },        "172.16.0.0": {          "end": "172.31.255.255",          "org": "MyOrg"        }      },      "known_orgs": {        "MyOrg": {          "name": "MyOrg",          "org": "MyOrg",          "descr": "MyOrg",          "cc": "US",          "country": "United States",          "city": "Anytown",          "state": "Somestate"        }      }    }  },  "plugins": {    "SNORT": "Info::Snort",    "WINDOWS": "Info::Windows",    "URL": "Info::Url",    "BRO_NOTICE": "Info::Bro"  },  "info": {    "snort": {      "url_templates": [        "http://doc.emergingthreats.net/bin/view/Main/%d"      ]    },    "url": {      "url_templates": [        "http://whois.domaintools.com/%s"      ]    },    "windows": {      "url_templates": [        "http://www.ultimatewindowssecurity.com/securitylog/encyclopedia/event.aspx?eventid=%d"      ]    }  },  "max_concurrent_archive_queries": 4,  "schedule_interval": 60,  "node_info_cache_timeout": 600,  "email": {    "display_address": "norepl...@example.com",    "base_url": "http://elsa/",    "subject": "ELSA Alert"  },  "link_key": "secret",  "yui": {    "local": "inc"  },  "meta_db": {    "dsn": "dbi:mysql:database=elsa_web",    "username": "elsa",    "password": "biglog"  },  "auth": {    "method": "security_onion"  },  "admin_groups": [    "system",    "admin"  ],  "auth_db": {    "dsn": "dbi:mysql:database=securityonion_db",    "username": "root",    "password": "",    "auth_statement": "SELECT PASSWORD(password) FROM user_info WHERE username=?",    "email_statement": "SELECT email FROM user_info WHERE username=?"  },  "peer_id_multiplier": 1000000000000,  "query_timeout": 100000,  "nodes": {    "127.0.0.1": {      "db": "syslog",      "username": "elsa",      "password": "biglog",      "port": 3306,      "sphinx_port": 9306    }  },  "pcap_url": "https://45.55.186.244/capme",  "logdir": "/nsm/elsa/data/elsa/log",  "buffer_dir": "/nsm/elsa/data/elsa/tmp/buffers",  "debug_level": "DEBUG",  "default_start_time_offset": 2,  "livetail": {    "poll_interval": 5,    "time_limit": 3600  }}' (JSON::ParserError)
        from /usr/lib/ruby/1.9.1/json/common.rb:148:in `parse'
        from /usr/bin/securityonion_elsa_register.rb:214:in `parse_conf'
        from /usr/bin/securityonion_elsa_register.rb:302:in `<main>'
Error updating /etc/elsa_web.conf for ELSA 1205.



Kevin

Doug Burks

unread,
Jul 3, 2015, 12:36:38 PM7/3/15
to securit...@googlegroups.com
On Fri, Jul 3, 2015 at 12:29 PM, Kevin Branch
<ke...@branchnetconsulting.com> wrote:
> It looks like the JSON config in my elsa_web.conf file was parse-able by
> ELSA rev 1090 but not by the "/usr/bin/securityonion_elsa_register.rb
> --migrate-web-1205" process. Is it possible that this process is assuming
> one and only one peer is defined in the "peers" section of the config? This
> server has multiple peers.

No, it should handle multiple peers just fine. Have you
double-checked for any JSON syntax errors?

Doug Burks

unread,
Jul 3, 2015, 12:55:30 PM7/3/15
to securit...@googlegroups.com
I tested your elsa_web.conf from your previous email with
securityonion_elsa_register.rb and it seems to have worked for me
(updated to 1205 format and preserved all peers).

You may want to just try re-running:
sudo /usr/bin/securityonion_elsa_register.rb --migrate-web-1205

If tthat doesn't work, perhaps there is some extra character somewhere
in your elsa_web.conf that was not included in the copy/paste.

Kevin Branch

unread,
Jul 3, 2015, 1:00:33 PM7/3/15
to securit...@googlegroups.com
Doug, 

I see that I did have a JSON syntax error that I inadvertently introduced shortly before I ran soup.  I commented one of the peers out of the peer section.  ELSA appears to run OK with lines commented out with the # symbol in elsa_web.conf but it appears this is technically illegal JSON syntax and /usr/bin/securityonion_elsa_register.rb  does not like it.  It appears that the script may not be stripping out the comments as intended, at least in certain cases.

As an after the fact workaround, I took a known working vanilla elsa_web.conf file from a freshly soup-updated SO system and manually incorporated my problem SO server's api key and peers section into it.  It works fine now.

Kevin


On Fri, Jul 3, 2015 at 12:36 PM, Doug Burks <doug....@gmail.com> wrote:

Kevin Branch

unread,
Jul 3, 2015, 1:03:55 PM7/3/15
to securit...@googlegroups.com
Sorry, that wasn't the right copy of elsa_web.conf.  My bad.  Here is the front end of the original file that the upgrade process choked on:

{
  "apikeys": {
    "elsa": "*REDACTED*"
  },
  "version": {
    "Author": "mcholste",
    "Date": "2013-12-04 12:00:00 -0400 (Wed, 04 Dec 2013)",
    "Rev": "1090",
    "Sphinx": "Sphinx 2.0.7-id64-dev (rel20-r373)"
  },
  "peers": {
    "127.0.0.1": {
      "url": "https://127.0.0.1:3154/",
      "username": "elsa",
      "apikey": "*REDACTED*"
    },
#    "CLIENT1": {
#      "url": "https://10.81.92.11:3154/",
#      "username": "elsa",
#      "apikey": "*REDACTED*"
#    },
    "CLIENT2": {
      "url": "https://10.81.92.15:3154/",
      "username": "elsa",
      "apikey": "*REDACTED*"
    },
#    "CLIENT3": {
#      "url": "https://10.81.92.13:3154/",
#      "username": "elsa",
#      "apikey": "*REDACTED*"
#    },
#    "CLIENT4": {
#      "url": "https://10.81.92.17:3154/",
#      "username": "elsa",
#      "apikey": "*REDACTED*"
#    }
  },




Reply all
Reply to author
Forward
0 new messages