Cannot access Moloch via HTTPS

[suc...@punchcyber.com]

I'm able to get to my moloch instance via hxxp://hostname:8005 but not hxxps://hostname:8005. The singlehost install script ran successfully and it created the self-signed certs. In addition, I was able to add users as well. The instance is running just fine collecting data and everything.

[Andy]

The node determines if it is https or not by looking for both the certFile and keyFile setting.  If they aren't both set then it doesn't start in https mode and does http.

If you can't figure it out gist your config file and your startup script.

[suc...@punchcyber.com]

I checked the config.ini file and both the certFile and keyFile are set to default; and the files are in their default location. I've listed the startup script below:

 

#!/bin/sh

TDIR=/data/moloch

cd ${TDIR}/elasticsearch-1.4.3
ulimit -a
# Uncomment if using Sun Java for better memory utilization
# export JAVA_OPTS="-XX:+UseCompressedOops"
export ES_HOSTNAME=`hostname -s`a

# Increase memory
ES_HEAP_SIZE=20G bin/elasticsearch -Des.config=${TDIR}/etc/elasticsearch.yml -d

 

sleep 5

 

#Start packet capture

# Add to /etc/inittab something like
# m1:2345:respawn:/data/moloch/bin/run_capture.sh

cd ${TDIR}/bin
/bin/rm -f ${TDIR}/logs/capture.log.old
/bin/mv ${TDIR}/logs/capture.log ${TDIR}/logs/capture.log.old

${TDIR}/bin/moloch-capture -c ${TDIR}/etc/config.ini > ${TDIR}/logs/capture.log$
#screen -d -RR CAPTURE ${TDIR}/bin/capture.cmd

 

sleep 5

 

#Start Moloch viewer

cd ${TDIR}/viewer
/bin/rm -f ${TDIR}/logs/viewer.log.old
/bin/mv ${TDIR}/logs/viewer.log ${TDIR}/logs/viewer.log.old
export NODE_ENV=production
exec ${TDIR}/bin/node viewer.js -c ${TDIR}/etc/config.ini > ${TDIR}/logs/viewer$

[Andy]

What does /data/moloch/etc/config.ini have in it?  Are you sure you didn't move the certFile and keyFile settings into another [section] by mistake

[suc...@punchcyber.com]

Yea; I didn't actually move anything in the config file. Here's what the config.ini file has in it:

 

# Cert file to use, comment out to use http instead
certFile=/data/moloch/etc/moloch.crt

# Private key file to use, comment out to use http instead
keyFile=/data/moloch/etc/moloch.key

[Andy]

Unfortunately I need to see the entire config file so I can repro, otherwise I just did a fresh install on ubuntu 12 with no issue.

Andy

[suc...@punchcyber.com]

I understand: Here's the config file.

 

[default]
# Comma seperated list of elasticsearch host:port combinations.  If not using a
# elasticsearch VIP, a different elasticsearch node in the cluster can be specified
# for each Moloch node to help spread load on high volume clusters
elasticsearch=127.0.0.1:9200

# How often to create a new elasticsearch index. hourly,daily,weekly,monthly
# Changing the value will cause previous sessions to be unreachable
rotateIndex=daily

# Cert file to use, comment out to use http instead
certFile=/data/moloch/etc/moloch.crt

# Private key file to use, comment out to use http instead
keyFile=/data/moloch/etc/moloch.key

# File with trusted roots/certs. WARNING! this replaces default roots
# Useful with self signed certs and can be set per node.
# caTrustFile=roots.cert

# S2S and Password Hash secret - Must be in default section. Since elasticsearch
# is wide open by default, we encrypt the stored password hashes with this
# so a malicous person can't insert a working new account.  It is also used
# for secure S2S communication. Comment out for no user authentication.
# Changing the value will make all previously stored passwords no longer work.
passwordSecret = MalwareHunter

# HTTP Digest Realm - Must be in default section.  Changing the value
# will make all previously stored passwords no longer work
httpRealm = Moloch

# The base url for Moloch web access.  Must end with a / or bad things will happen
# Default: "/"
# webBasePath = /moloch/

# The interface to listen on for traffic
interface=eth2

# The bpf filter
#bpf=

# The yara file name
#yara=

# Uncomment to log access requests to a different log file
#accessLogFile = /data/moloch/logs/access.log

 

# The directory to save raw pcap files to
pcapDir = /data/moloch/raw

# The max raw pcap file size in gigabytes, with a max value of 36G.
# The disk should have room for at least 10*maxFileSizeG
maxFileSizeG = 1

# The max time in minutes between rotating pcap files.  Default is 0, which means
# only rotate based on current file size and the maxFileSizeG variable
#maxFileTimeM = 60

# TCP timeout value.  Moloch writes a session record after this many seconds
# of inactivity.
tcpTimeout = 600

# Moloch writes a session record after this many seconds, no matter if
# active or inactive
tcpSaveTimeout = 720

# UDP timeout value.  Moloch assumes the UDP session is ended after this
# many seconds of inactivity.
udpTimeout = 30

# ICMP timeout value.  Moloch assumes the ICMP session is ended after this
# many seconds of inactivity.
icmpTimeout = 10

# An aproximiate maximum number of active sessions Moloch/libnids will try
# and monitor
maxStreams = 1000000

# Moloch writes a session record after this many packets
maxPackets = 10000

# Delete pcap files when free space is lower then this in gigabytes OR it can be
# expressed as a percentage (ex: 5%).  This does NOT delete the session records in
# the database. It is recommended this value is between 5% and 10% of the disk.
# Database deletes are done by the db.pl expire script
freeSpaceG = 600

# The port to listen on, by default 8005
viewPort = 8005

# The host/ip to listen on, by default 0.0.0.0 which is ALL
#viewHost = 127.0.0.1

# By default the viewer process is https://hostname:<viewPort> for each node.
viewUrl = https://HOSTNAME:8005

# Path of the maxmind geoip country file.  Download free version from:
#  http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
geoipFile = /data/moloch/etc/GeoIP.dat

# Path of the maxmind geoip ASN file.  Download free version from:
#  http://www.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz
geoipASNFile = /data/moloch/etc/GeoIPASNum.dat

# Path of the rir assignments file
#  https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.csv
rirFile = /data/moloch/etc/ipv4-address-space.csv

# User to drop privileges to. The pcapDir must be writable by this user or group below
dropUser=root

# Group to drop privileges to. The pcapDir must be writable by this group or user above
dropGroup=root

# Semicolon ';' seperated list of tags which once capture sets for a session causes the
# remaining pcap from being saved for the session.  It is likely that the initial packets
# WILL be saved for the session since tags usually aren't set until after several packets
# Each tag can optionally be followed by a :<num> which specifies how many total packets to save
#dontSaveTags=

# Header to use for determining the username to check in the database for instead of
# using http digest.  Use this if apache or something else is doing the auth.
# Might need something like this in the httpd.conf
# RewriteRule .* - [E=ENV_RU:%{REMOTE_USER}]
# RequestHeader set MOLOCH_USER %{ENV_RU}e
#userNameHeader=moloch_user

# Should we parse extra smtp traffic info
parseSMTP=true

# Should we parse extra smb traffic info
parseSMB=true

# Should we parse HTTP QS Values
parseQSValue=false

# Semicolon ';' seperated list of SMTP Headers that have ips, need to have the terminating colon ':'
smtpIpHeaders=X-Originating-IP:;X-Barracuda-Apparent-Source-IP:

# Semicolon ';' seperated list of directories to load parsers from
parsersDir=/data/moloch/parsers

# Semicolon ';' seperated list of directories to load plugins from
pluginsDir=/data/moloch/plugins

# Semicolon ';' seperated list of plugins to load and the order to load them
#plugins=tagger.so; netflow.so

# Semicolon ';' seperated list of viewer plugins to load and the order to load in
# viewerPlugins=wise.js

# NetFlowPlugin
# Input device id, 0 by default
#netflowSNMPInput=1
# Outout device id, 0 by default
#netflowSNMPOutput=2
# Netflow version 1,5,7 supported, 7 by default
#netflowVersion=1
# Semicolon ';' seperated list of netflow destinations
#netflowDestinations=127.0.0.1:9993

# Specify the max number of indices we calculate spidata for.
# ES will blow up if we allow the spiData to search too many indices.
spiDataMaxIndices=3

# Uncomment the following to allow direct uploads.  This is experimental
#uploadCommand=/data/moloch/bin/moloch-capture --copy -n {NODE} -r {TMPFILE} -c {CONFIG} {TAGS}

# Title Template
# _cluster_ = ES cluster name
# _userId_  = logged in User Id
# _userName_ = logged in User Name
# _page_ = internal page name
# _expression_ = current search expression if set, otherwise blank
# _-expression_ = " - " + current search expression if set, otherwise blank, prior spaces removed
# _view_ = current view if set, otherwise blank
# _-view_ = " - " + current view if set, otherwise blank, prior spaces removed
#titleTemplate=_cluster_ - _page_ _-view_ _-expression_

# ADVANCED - Semicolon ';' seperated list of files to load for config.  Files are loaded
# in order and can replace values set in this file or previous files.
#includes=

# ADVANCED - How is pcap written to disk
#  normal        = just normal open/writes uses the linux page cache heavily (default)
#                  probably want to tune vm.dirty_ratio and vm.dirty_background_ratio
#  direct        = use O_DIRECT with open/write - writes directly to drives, no page cache, increase
#                  pcapWriteSize (>= 256k, must be multiple of 4096) and packetsPerPoll (>= 100k)
#  thread        = like normal, but use a thread for all the writes
#  thread-direct = like direct, but use a thread for all the writes
pcapWriteMethod=normal

# ADVANCED - Buffer size when writing pcap files.  Should be a multiple of the raid 5 or xfs
# stripe size.  Defaults to 256k
pcapWriteSize = 262143

# ADVANCED - value for pcap_set_buffer_size, may not be used depending on kernel etc
pcapBufferSize = 30000000

# ADVANCED - Number of bytes to bulk index at a time
dbBulkSize = 300000

# ADVANCED - Number of seconds before we force a flush to ES
dbFlushTimeout = 5

# ADVANCED - Compress requests to ES, reduces ES bandwidth by ~80% at the cost
# of increased CPU. MUST have "http.compression: true" in elasticsearch.yml file
compressES = false

# ADVANCED - Max number of connections to elastic search
maxESConns = 30

# ADVANCED - Max number of es requests outstanding in q
maxESRequests = 500

# ADVANCED - Number of packets to ask libnids/libpcap to read per poll/spin
# Increasing may hurt stats and ES performance
# Decreasing may cause more dropped packets
packetsPerPoll = 50000

# ADVANCED - Moloch will try to compensate for SYN packet drops by swapping
# the source and destination addresses when a SYN-acK packet was captured first.
# Probably useful to set it false, when running Moloch in wild due to SYN floods.
antiSynDrop = true

# DEBUG - Write to stdout info every X packets.
# Set to -1 to never log status
logEveryXPackets = 100000

# DEBUG - Write to stdout unknown protocols
logUnknownProtocols = false

# DEBUG - Write to stdout elastic search requests
logESRequests = true

# DEBUG - Write to stdout file creation information
logFileCreation = true

##############################################################################
# Classes of nodes
# Can override most default values, and create a tag call node:<classname>
[test]
freeSpaceG = 200

##############################################################################
# Nodes
# Usually just use the hostname before the first dot as the node name
# Can override most default values

[moloch-node01]
nodeClass = test
# Might use a different elasticsearch node
####elasticsearch=es-moloch-test

#[class2]
#nodeClass = daha
# Might use a different elasticsearch node
#elasticsearch=elasticsearchhost2
# Uses a different interface
#interface = eth4

##############################################################################
# override-ips is a special section that overrides the MaxMind databases for
# the fields set, but fields not set will still use MaxMind (example if you set
# tags but not country it will use MaxMind for the country)
# Spaces and capitalization is very important.
# IP Can be a single IP or a CIDR
# Up to 10 tags can be added
#
# ip=tag:TAGNAME1;tag:TAGNAME2;country:3LetterUpperCaseCountry;asn:ASN STRING
#[override-ips]
#10.1.0.0/16=tag:ny-office;country:USA;asn:AS0000 This is an ASN

##############################################################################
# It is now possible to define in the config file extra http/email headers
# to index.  They are accessed using the expression http.<fieldname> and
# email.<fieldname> with optional .cnt expressions
#
# Possible config atributes for all headers
#   type:<string> (string|integer|ip)  = data type                (default string)
#  count:<boolean>                     = index count of items     (default false)
#  unique:<boolean>                    = only record unique items (default true)

# headers-http-request is used to configure request headers to index
#[headers-http-request]
#referer=type:string;count:true;unique:true

# headers-http-response is used to configure http response headers to index
#[headers-http-response]
#location=type:string;count:true

# headers-email is used to configure email headers to index
#[headers-email]
#x-priority=type:integer

[Andy]

I tried on fresh install VM and it worked fine

   curl --insecure https://localhost:8005

returns Unauthorized.

Not sure how to reproduce.  You could try editing viewer.js and towards the very bottom you will see

if (Config.isHTTPS()) {

change to

console.log(Config.isHTTPS() != undefined, Config.get("certFile"), Config.get("keyFile"));

if (Config.isHTTPS()) {

The output should be

true '/data/moloch/etc/moloch.crt' '/data/moloch/etc/moloch.key'

[suc...@punchcyber.com]

I'm able to use https now but moloch does not ask for a login. When I access moloch via https://localhost:8005 I get taken straight to the sessions page; no authentication required. Any thoughts on how to fix it?

[Andy]

What did you change to make https work?  Do you have passwordSecret set?

[suc...@punchcyber.com]

Removed the comments and additional white space that was around the certFile and keyFile definitions in the config.ini. I do have passwordSecret set.

[Andy]

that shouldn't matter.  You want want to try going into your viewer directory and do the following

mv node_modules node_modules.save

npm cache clean   # Make sure there are no error, might have to run as root

npm install

[suc...@punchcyber.com]

I tried the steps you suggested; and there were no erros when i did the npm cache clean; however now the viewer won't load. I checked the log file and it says:

 

'ERROR - Couldn't load some dependancies, maybe need to 'npm update' inside viewer directory { [Error: Cannot find module 'express'] code: 'MODULE_NOT_FOUND' }'

 

I tried 'npm install express' and I got the following output:

npm WARN package.json mol...@0.11.0 No description
npm WARN package.json mol...@0.11.0 No repository field.
npm WARN package.json mol...@0.11.0 No README data
exp...@3.20.2 node_modules/express
├── basic...@1.0.0
├── merge-de...@1.0.0
├── utils...@1.0.0
├── cookie-s...@1.0.6
├── met...@1.1.1
├── coo...@0.1.2
├── fr...@0.2.4
├── escap...@1.0.1
├── range-...@1.0.2
├── conten...@1.0.1
├── va...@1.0.0
├── pars...@1.3.0
├── content-d...@0.5.0
├── comm...@2.6.0
├── de...@1.0.1
├── de...@2.1.3 (m...@0.7.0)
├── et...@1.5.1 (c...@3.2.1)
├── mkd...@0.5.0 (mini...@0.0.8)
├── se...@0.12.2 (des...@1.0.3, m...@0.7.0, mi...@1.3.4, on-fi...@2.2.0)
├── proxy...@1.0.7 (forw...@0.1.0, ipad...@0.1.9)
└── con...@2.29.1 (pa...@0.0.1, respon...@2.3.0, vh...@3.0.0, on-he...@1.0.0, by...@1.0.0, basic-aut...@1.0.0, cookie...@1.3.4, method-...@2.3.2, serve-...@1.9.2, connect...@1.6.1, q...@2.4.1, http-...@1.3.1, serve-...@2.2.0, finalh...@0.3.4, mor...@1.5.2, express...@1.10.4, cs...@1.7.0, typ...@1.6.1, body-...@1.12.3, errorh...@1.3.5, compr...@1.4.3, serve...@1.6.3, multi...@3.3.2)

All the modules appear to be installed.

[Andy]

On Tuesday, April 21, 2015 at 11:11:30 AM UTC-4, suc...@punchcyber.com wrote:

I tried the steps you suggested; and there were no erros when i did the npm cache clean; however now the viewer won't load. I checked the log file and it says:

 

'ERROR - Couldn't load some dependancies, maybe need to 'npm update' inside viewer directory { [Error: Cannot find module 'express'] code: 'MODULE_NOT_FOUND' }'

 

I tried 'npm install express' and I got the following output:

That installs express, so what happens now when you run? :) 

[suc...@punchcyber.com]

I'm sorry I should've clarified; even after I run "npm install express" the viewer.log still shows same error.

[Andy]

Are you sure it didn't change to another package that isn't installed?

Did you try just "npm install" again or "npm update"?

Can you provide "ls -l /data/moloch/viewer/node_modules"  (assuming /data/moloch is where you run from)

[suc...@punchcyber.com]

I ran npm install

 

here's the output of ls -l

drwx------. 3 suchit.vaidya unixusers 8192 Apr 21 14:51 async
drwx------. 4 suchit.vaidya unixusers 8192 Apr 21 14:51 elasticsearch
drwx------. 5 suchit.vaidya unixusers 8192 Apr 21 15:10 express
drwx------. 4 suchit.vaidya unixusers 8192 Apr 21 14:51 fs-ext
drwx------. 4 suchit.vaidya unixusers   82 Apr 21 14:44 iniparser
drwx------. 5 suchit.vaidya unixusers 8192 Apr 21 14:51 jade
drwx------. 3 suchit.vaidya unixusers 8192 Apr 21 14:44 keep-alive-agent
drwx------. 3 suchit.vaidya unixusers 8192 Apr 21 15:05 minimatch
drwx------. 4 suchit.vaidya unixusers 8192 Apr 21 14:51 moment
drwx------. 4 suchit.vaidya unixusers  108 Apr 21 14:44 passport
drwx------. 4 suchit.vaidya unixusers 8192 Apr 21 14:44 passport-http
drwx------. 8 suchit.vaidya unixusers 8192 Apr 21 14:44 png
drwx------. 4 suchit.vaidya unixusers 8192 Apr 21 14:50 semver
drwx------. 5 suchit.vaidya unixusers 8192 Apr 21 14:51 stylus

[Andy]

Permissions are bad.   You can either wipe node_modules again and make sure you do "umask 022" before the npm installs, or you could do something like

find /data/moloch -type d -exec chmod og+rx {} \;

find /data/moloch -type f -exec chmod og+r {} \;