Connectivity Issue

[BBCan177]

On Sunday, January 26, 2014 4:03:56 PM UTC-5, Doug Burks wrote:
> On Sat, Jan 25, 2014 at 2:57 PM, BBCan177 <bbca...@gmail.com> wrote:
>
> > I have updated my SO server without any issues, but I am having issue with a remote sensor.
>
> >
>
> >
>
> > The server was updated first, reboot, than the sensor was updated and a reboot.
>
> >
>
> > When I ran sudo rule-update on the sensor -
>
> >
>
> > ssh: connect to host xx.xx.xx.xx port xxx: No route to host
>
> >
>
> >
>
> > I cant ping or ssh from sensor to server or server to sensor.
>
> > I can ping the local router but not the remote router(Ipsec VPN tunnel)(From the sensor). Pinging 8.8.8.8 or google.com works fine.
>
> >
>
> > However, I can ssh to the sensor from other Devices. All other network devices seem to be working fine.
>
> >
>
> > UFW is set properly on both sides.
>
> >
>
> > Was working prior to updates. Not sure if one of the other updates affected the settings.
>
>
>
> Hi BBCan177,
>
>
>
> I don't believe any of our updates would have caused this issue. It
>
> sounds like it's more likely an issue with your network (perhaps ACLs
>
> on your router?) or with some custom UFW settings. Have you made any
>
> manual changes to UFW? If so, please start a new thread and include
>
> the output of "sudo ufw status" from both server and sensor.

Hi Doug,

Thanks for your help.

The sensor has been running for 80 days without any issues. As soon as I completed the last round of updates it went off course from there. Maybe it was one of the Ubuntu updates?

I have checked my Switches, Router and "MS Server 2008 AD/DNS" but cant find the culprit yet? I made no changes to UFW or any other tweaks for that matter recently.

Setting

sysctl -w net.ipv4.ip_forward=1

seems to fix the issue until I can dig deeper into this issue. Pings are now responding with:

From xx.xx.xx.xx: icmp_seq=1 Redirect Host(New nexthop: xx.xx.xx.xx)
64 bytes from xx.xx.xx.xx: icmp_req=1 ttl=62 time=49.7 ms

Can anyone share their experience with setting an Ubuntu machine in MS Server 2008 RC2 AD/DNS?

Does the hostname /etc/host and /etc/hostname need any special consideration?
In /etc/network/interfaces, are there any special requirements for compatibility to MS Server DNS?

[Doug Burks]

Replies inline.

On Mon, Jan 27, 2014 at 11:09 AM, BBCan177 <bbca...@gmail.com> wrote:
> The sensor has been running for 80 days without any issues. As soon as I completed the last round of updates it went off course from there. Maybe it was one of the Ubuntu updates?

I doubt that Ubuntu updates would cause this.

> I have checked my Switches, Router and "MS Server 2008 AD/DNS" but cant find the culprit yet? I made no changes to UFW or any other tweaks for that matter recently.

Have you ever made any changes to UFW, or are you running our default UFW rules?

> Setting
>
> sysctl -w net.ipv4.ip_forward=1
>
> seems to fix the issue until I can dig deeper into this issue.

Not sure why you would need to set that if you're running our default UFW rules.

> Pings are now responding with:
>
> From xx.xx.xx.xx: icmp_seq=1 Redirect Host(New nexthop: xx.xx.xx.xx)
> 64 bytes from xx.xx.xx.xx: icmp_req=1 ttl=62 time=49.7 ms

Based on your ping output, I wonder if your UFW rules may have been
blocking ICMP (redirects)?

> Can anyone share their experience with setting an Ubuntu machine in MS Server 2008 RC2 AD/DNS?

I don't think this would be related to your issue.

> Does the hostname /etc/host and /etc/hostname need any special consideration?

I don't think this would be related to your issue.

> In /etc/network/interfaces, are there any special requirements for compatibility to MS Server DNS?

I don't think this would be related to your issue.

Please send the output of "sudo ufw status" from both server and sensor.

--
Doug Burks

[BBCan177]

Ping and SSH dont work unless i run that sysctl command. I believe I tried to tested a sensor fully open rule for UFW on the server without any success.

--SERVER--

sudo ufw status
Status: active

To Action From
-- ------ ----
22/tcp ALLOW x.x.x.x
22/tcp ALLOW x.x.x.x (Sensor ip)
514 ALLOW x.x.x.x
514 ALLOW x.x.x.x
514 ALLOW x.x.x.x
514 ALLOW x.x.x.x
514 ALLOW x.x.x.x
514 ALLOW x.x.x.x
514 ALLOW x.x.x.x
1514/udp ALLOW x.x.x.x
1514/udp ALLOW x.x.x.x
1514/udp ALLOW x.x.x.x
1514/udp ALLOW x.x.x.x
1514/udp ALLOW x.x.x.x
1514/udp ALLOW x.x.x.x
1514/udp ALLOW x.x.x.x
1514/udp ALLOW x.x.x.x
1514/udp ALLOW x.x.x.x
4505,4506/tcp ALLOW x.x.x.x (Sensor ip)
7736/tcp ALLOW x.x.x.x (Sensor ip)
3389/tcp ALLOW x.x.x.x
3389/tcp ALLOW x.x.x.x

--SENSOR--

sudo ufw status
Status: active

To Action From
-- ------ ----
22/tcp ALLOW x.x.x.x
22/tcp ALLOW x.x.x.x (Server ip)
514 ALLOW x.x.x.x (Server ip)
3389/tcp ALLOW x.x.x.x
3389/tcp ALLOW x.x.x.x
1514/udp ALLOW x.x.x.x (ossec agent)
1514/udp ALLOW x.x.x.x (ossec agent)

(Probably dont need the 514 on the sensor?)

[Doug Burks]

On Mon, Jan 27, 2014 at 12:01 PM, BBCan177 <bbca...@gmail.com> wrote:
> Ping and SSH dont work unless i run that sysctl command.

I still don't understand why you would need that. "sysctl -w
net.ipv4.ip_forward=1" enables IP forwarding to allow the Linux kernel
to route packets between interfaces. You just have one management
interface in each box, right?

> I believe I tried to tested a sensor fully open rule for UFW on the server without any success.

Yeah, based on the output below, you've definitely made changes to our
default UFW rules. Have you made any other firewall changes, perhaps
directly using IPTables?

Is it possible OSSEC Active Response is creating block rules? Run the
following command on both boxes and see if you see any interesting IP
address being blocked:
sudo iptables -nvL

Restarting OSSEC should remove any blocks:
sudo service ossec-hids-server restart

You're right, your sensor probably doesn't need to receive syslog from
your server IP.

--
Doug Burks

[BBCan177]

> I still don't understand why you would need that. "sysctl -w
>
> net.ipv4.ip_forward=1" enables IP forwarding to allow the Linux kernel
>
> to route packets between interfaces. You just have one management
>
> interface in each box, right?
>

I didn't make any changes to iptables except for the UFW ones.

Yes only one management port, but this sensor has a two sensor NICS. Only one is being used.

See attached iptables for both machines.

-- SENSOR--

ifconfig

eth0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:xx.xx.xx.xx Bcast:xx.xx.xx.255 Mask:255.255.255.0
inet6 addr: xxxx::xxx:xxxx:xxxx:xxxx/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:750659 errors:0 dropped:0 overruns:0 frame:0
TX packets:460220 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:122675477 (122.6 MB) TX bytes:409259079 (409.2 MB)
Interrupt:19 Memory:f0180000-f01a0000

eth1 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
UP BROADCAST RUNNING NOARP PROMISC MULTICAST MTU:1500 Metric:1
RX packets:4857110 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1017130172 (1.0 GB) TX bytes:0 (0.0 B)
Interrupt:16 Memory:f0280000-f02a0000

eth2 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
UP BROADCAST NOARP PROMISC MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:16 Memory:f0300000-f0320000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:554233 errors:0 dropped:0 overruns:0 frame:0
TX packets:554233 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:783922279 (783.9 MB) TX bytes:783922279 (783.9 MB)

>
>
> Is it possible OSSEC Active Response is creating block rules?
>

I usually run this command, and it didnt have any blocks.

iptables -L INPUT -n -v

--SERVER--

sudo iptables -L INPUT -n -v

Chain INPUT (policy DROP 34 packets, 1584 bytes)
pkts bytes target prot opt in out source destination
4666K 5153M ufw-before-logging-input all -- * * 0.0.0.0/0 0.0.0.0/0
4666K 5153M ufw-before-input all -- * * 0.0.0.0/0 0.0.0.0/0
64647 8908K ufw-after-input all -- * * 0.0.0.0/0 0.0.0.0/0
34 1584 ufw-after-logging-input all -- * * 0.0.0.0/0 0.0.0.0/0
34 1584 ufw-reject-input all -- * * 0.0.0.0/0 0.0.0.0/0
34 1584 ufw-track-input all -- * * 0.0.0.0/0 0.0.0.0/0

Here is the route and arp with the

sysctl -w net.ipv4.ip_forward=0.

As this is a remote location the arp for SO1 (SO Server) shouldnt be listed as its not a local machine. But I read somewhere that if it cant find a host it can list it as (Incomplete)

route -n

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 xx.xx.xx.xx 0.0.0.0 UG 100 0 0 eth0
xx.xx.xx.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 eth0

arp

Address HWtype HWaddress Flags Mask Iface
as01.xxx.domain.com ether xx:xx:xx:xx:xx:xx C eth0
xx.xx.xx.xx ether xx:xx:xx:xx:xx:xx C eth0
so1.doman.com (incomplete) eth0

DIG -x and nslookups both respond with the correct details of the SO Server

-- SENSOR--

dig -x xx.xx.xx.xx (SO SERVER)

; <<>> DiG 9.8.1-P1 <<>> -x xx.xx.xx.xx
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36676
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;xx.xx.xx.xx.in-addr.arpa. IN PTR

;; ANSWER SECTION:
xx.xx.xx.xx.in-addr.arpa. 3600 IN PTR so1.domain.com.

;; Query time: 1 msec
;; SERVER: 10.1.40.7#53(xx.xx.xx.xx) (**MS DNS SERVER**)
;; WHEN: Mon Jan 27 18:03:04 2014
;; MSG SIZE rcvd: 71

-------------------------------------

So basically,

the SO Server 10.10.1.4
the SO Sensor 10.10.2.4

From the Sensor (10.10.2.4) I cant reach anything on the 10.10.1.0/24 network without the sysctl command being set to "1".

However, I can connect from 10.10.2.4 to anything on the 10.10.3.0/24 (another VPN IPSEC tunnel) without issues.

[Doug Burks]

Are you running a VPN client or something else that would create a
virtual network interface? That's the only reason I can think of that
setting "sysctl -w net.ipv4.ip_forward=1" would make things work.

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/groups/opt_out.



--
Doug Burks

[BBCan177]

Thanks for your Patience Doug,

There is no Virtual VPN Interface or anything similar that I can think of? Wouldn't ifconfig show all interfaces?

When I set the "sysctl -w net.ipv4.ip_forward=0", I flush DNS, Restart Networking and the first ping will go thru with a response but the rest are Host not Reachable.

I will try to run a command to get a complete firewall table listing. pfSense also has a packet capture that I will try.

-----
This doesn't make sense either? But SGUIL shows the status as "UP" with it set as "1"

ping xx.xx..xx.xx (Ping from SO Sensor to SO Server)

PING xx.xx.xx.xx (xx.xx.xx.xx) 56(84) bytes of data.

From xx.xx.xx.xx: icmp_seq=1 Redirect Host(New nexthop: xx.xx.xx.xx)

64 bytes from 10.1.34.3: icmp_req=1 ttl=63 time=46.8 ms

traceroute xx.xx.xx.xx (Traceroute from SO Sensor to SO Server)

traceroute to xx.xx.xx.xx (xx.xx.xx.xx), 30 hops max, 60 byte packets
1 xx.xx.xx.xx (xx.xx.xx.xx) 0.545 ms 0.532 ms 0.519 ms
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *

(Ping/Traceroute from SO Server to SO Sensor)

Same response as above but doesn't show the "Redirect Host" in the ping response.

[Doug Burks]

I'd take Sguil and everything else out of the equation and focus on
good ole ping for troubleshooting. With the default ufw rules, you
should be able to ping from the server to the sensor and from the
sensor to the server. If you can't, then there's a problem in your

network. Previously you said:

"From the Sensor (10.10.2.4) I cant reach anything on the 10.10.1.0/24
network without the sysctl command being set to "1".
However, I can connect from 10.10.2.4 to anything on the 10.10.3.0/24
(another VPN IPSEC tunnel) without issues."

Based on that, I'd say your 10.10.3.0/24 network is functioning
correctly, but your 10.10.1.0/24 is not.

I'd recommend running a packet capture on the management interface of
your sensor. The pcap should show you exactly where things are
breaking down.

[BBCan177]

Hi Doug,

I captured some traffic and was able to figure out that the Router was not passing traffic properly. I had made some changes to the router to get my routers syslogs thru the vpn tunnel as follows:

https://doc.pfsense.org/index.php/Why_can't_I_query_SNMP,_use_syslog,_NTP,_or_other_services_initiated_by_the_firewall_itself_over_IPsec_VPN%3F

Once I removed the manual route and rebooted the router, the SO Server and Sensor came back to life. Unfortunately, the remote syslogs arent making it to ELSA now.

In regards to why the "sysctl -w net.ipv4.ip_forward=1" was working, please see the attached document. It seems to have used NTP? Could this be used as a DDOS attack?

Thanks, as always, for your help.

[Doug Burks]

Replies inline.

On Tue, Jan 28, 2014 at 4:53 PM, BBCan177 <bbca...@gmail.com> wrote:
> Hi Doug,
>
> I captured some traffic and was able to figure out that the Router was not passing traffic properly. I had made some changes to the router to get my routers syslogs thru the vpn tunnel as follows:
>
> https://doc.pfsense.org/index.php/Why_can't_I_query_SNMP,_use_syslog,_NTP,_or_other_services_initiated_by_the_firewall_itself_over_IPsec_VPN%3F
>
> Once I removed the manual route and rebooted the router, the SO Server and Sensor came back to life.

Glad you figured it out!

> Unfortunately, the remote syslogs arent making it to ELSA now.

Instead of sending syslog across the VPN tunnel, why not send it to
its local ELSA instance? That's one reason for our distributed ELSA
architecture: to keep data as close to its point of origin as
possible.

> In regards to why the "sysctl -w net.ipv4.ip_forward=1" was working, please see the attached document.

My guess is that ip_forward had something to do with this statement
from the pfsense link you sent:
"There's an :annoying but mostly harmless side-effect to this - every
LAN packet to the tunnel elicits a no-change ICMP Redirect."

> It seems to have used NTP?

Yes, Ubuntu uses NTP to synchronize its clock.

> Could this be used as a DDOS attack?

There have been some recent NTP DDOS attacks
(http://blog.cloudflare.com/understanding-and-mitigating-ntp-based-ddos-attacks),
but I'm not sure what that has to do with your pfsense situation.

--
Doug Burks

[BBCan177]

> > Unfortunately, the remote syslogs arent making it to ELSA now.
>
> Instead of sending syslog across the VPN tunnel, why not send it to
>
> its local ELSA instance? That's one reason for our distributed ELSA
>
> architecture: to keep data as close to its point of origin as
>
> possible.

Hi Doug,

I added the pfsense parser to the sensors patterndb.xml file, ran "service syslog-ng restart"

Opened up 514 tcp/udp on the sensors UFW firewall to allow the router access.
Set the router to send syslogs to the sensors IP.

But I am not seeing any events in the Servers ELSA web interface.

I searched with "class=FIREWALL_ACCESS_DENY groupby:host" and also tried to look for it by the SRC ip blocked at the router (class=none). I see other host events as those hosts are linked directly to the server.

Is there anything else to set on the server/sensor? I've looked at the wiki but couldn't find any additional information.

[Heine Lysemose]

Hi

Have you enabled pfSense to send syslog to the ELSA IP? And also enabled some rules to trigger syslog messages?

I am not in front of a terminal now so I can't point more directly right now.

Regards,
Lysemose

[BBCan177]

>
> Have you enabled pfSense to send syslog to the ELSA IP? And also enabled some rules to trigger syslog messages?
>
> I am not in front of a terminal now so I can't point more directly right now.
>

Thanks Heine,

Yes I set pfSense to point to the sensors IP. I also turned on "Log packets blocked by the default rule" so that it is sending a lot more alerts.

But nothing is showing up in ELSA yet.

[Heine Lysemose]

Any firewalls between pfSense and ELSA other than pfSense itself...

[BBCan177]

All on the same local network with no additional firewalls between.

[BBCan177]

sudo tcpdump -nnvvAi eth1 -s0 | grep "xx.xx.xx.gw"

tcpdump: WARNING: eth1: no IPv4 address assigned
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes

xx.xx.xx.gw.514 > xx.xx.xx.SO.514: [udp sum ok] SYSLOG, length: 275
xx.xx.xx.gw.514 > xx.xx.xx.SO.514: [udp sum ok] SYSLOG, length: 78
xx.xx.xx.gw.514 > xx.xx.xx.SO.514: [udp sum ok] SYSLOG, length: 236
xx.xx.xx.gw.514 > xx.xx.xx.SO.514: [udp sum ok] SYSLOG, length: 243
xx.xx.xx.gw.514 > xx.xx.xx.SO.514: [udp sum ok] SYSLOG, length: 364
xx.xx.xx.gw.514 > xx.xx.xx.SO.514: [udp sum ok] SYSLOG, length: 310
xx.xx.xx.gw.514 > xx.xx.xx.SO.514: [udp sum ok] SYSLOG, length: 220
xx.xx.xx.gw.514 > xx.xx.xx.SO.514: [udp sum ok] SYSLOG, length: 227
xx.xx.xx.gw.514 > xx.xx.xx.SO.514: [udp sum ok] SYSLOG, length: 364
xx.xx.xx.gw.514 > xx.xx.xx.SO.514: [udp sum ok] SYSLOG, length: 238
xx.xx.xx.gw.514 > xx.xx.xx.SO.514: [udp sum ok] SYSLOG, length: 91
xx.xx.xx.gw.514 > xx.xx.xx.SO.514: [udp sum ok] SYSLOG, length: 310
xx.xx.xx.gw.514 > xx.xx.xx.SO.514: [udp sum ok] SYSLOG, length: 238

[BBCan177]

pfSense has a patch to fix the syslog issue over VPN's

https://redmine.pfsense.org/projects/pfsense/repository/revisions/53c5407e646028a003b2765a87dd3316b21a9497

"This option will allow the logging daemon to bind to a single IP address, rather than all IP addresses."

The Remote Router is now pointed thru the vpn tunnel to the SO Server and is working.

I would still like to get it to point to the Sensors ELSA instance if possible?

[Doug Burks]

Are you sure the sensor has ELSA enabled?

Are you able to query the ELSA web interface on the master server and
get any other kinds of logs from that sensor?

Are you sure you allowed port 514 UDP in the sensor's firewall?

Are there any other hosts sending syslog over 514 UDP to that sensor?

After applying the pfSense patch to fix the syslog issue, have you
tried again pointing it to the local sensor?

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/groups/opt_out.

--
Doug Burks

[BBCan177]

Hi Doug,

> Are you sure the sensor has ELSA enabled?

Securityonion.conf (Remote Sensor)

ELSA=YES
BRO=YES

in Sensor.conf (Remote Sensor)

PCAP_ENABLED="yes"
PCAP_AGENT_ENABLED="yes"
SNORT_AGENT_ENABLED="yes"
IDS_ENGINE_ENABLED="yes"
BARNYARD2_ENABLED="yes"
PRADS_ENABLED="yes"
SANCP_AGENT_ENABLED="yes"
PADS_AGENT_ENABLED="yes"
ARGUS_ENABLED="yes"
HTTP_AGENT_ENABLED="yes"

> Are you able to query the ELSA web interface on the master server and
> get any other kinds of logs from that sensor?

I can query the Server ELSA web interface, but what is strange is that for the "Remote Sensor", the Conn logs and the Http logs only show connections/traffic for its local network only. Nothing logged for external sites or addresses?

The devices on the "Server Sensor" have both internal and external data.

I can see the local and external activity in SGUIL for alerts from the "Server Sensor" and from the "Remote Sensor". I can pivot from SGUIL on the "Server Sensor" alerts into ELSA but when I try to pivot on the "Remote Sensor Data" it doesn't report any events in ELSA. SGUIL is also showing all of the OSSEC alerts for both Server/and Remote Sensor.

> Are you sure you allowed port 514 UDP in the sensor's firewall?

YES

> Are there any other hosts sending syslog over 514 UDP to that sensor?

NO

> After applying the pfSense patch to fix the syslog issue, have you
> tried again pointing it to the local sensor?

Yes, after applying the patch, I left the syslog pointing to the sensor. From a previous post above, you can see the tcpdump output showing that 514/udp data was making it to the Sensor. When I point it to the Server, ELSA picks up the data.

I also have OSSEC Agents at this remote location and they are pointing to the sensor. I am getting Email alerts and the daily summary showing all the activity, but When I query any OSSEC data on the ELSA Server for any "Remote" OSSEC events it doesn't show any data. Any data that does show it related to "local" OSSEC agents reporting on the local network in regards to its activity with the "Remote" machines. The OSSEC data is reporting into SGUIL for both local and remote agents.

Looks like ELSA is broke somewhere in my installation?

[Doug Burks]

It sounds like perhaps the ELSA web interface is not querying the ELSA
log node on your sensor at all.

When you log into the ELSA web interface, how many node(s) does it
show in the upper right corner?

Please send your /etc/elsa_web.conf from the master server.

Double-check that the apikey for the sensor matches between the
sensor's elsa_web.conf and the server's elsa_web.conf.

[BBCan177]

> When you log into the ELSA web interface, how many node(s) does it
> show in the upper right corner?

Its only showing 1 Node

> Please send your /etc/elsa_web.conf from the master server.

See attached

> Double-check that the apikey for the sensor matches between the
> sensor's elsa_web.conf and the server's elsa_web.conf.

Looks like the APIs match? I can ping SO2 from the server.

[BBCan177]

Here is the correct Server elsa_web.conf

[Doug Burks]

On Sat, Feb 1, 2014 at 10:03 AM, BBCan177 <bbca...@gmail.com> wrote:
>
>> When you log into the ELSA web interface, how many node(s) does it
>> show in the upper right corner?
>
> Its only showing 1 Node

Is starman running on the sensor?
pgrep -lf starman

Is autossh running on the sensor?
pgrep -lf autossh

Have you tried restarting autossh on the sensor?
sudo pkill -USR1 autossh

Do you see the ssh tunnel on the server?
pgrep -lf sshd

Have you tried restarting Apache on the server?
sudo service apache2 restart


--
Doug Burks

[BBCan177]

Hi Doug,

I ran the commands as suggested. Autossh was running, but killing the autossh restored the second NODE in ELSA Web Interface.

I pointed the pfSense syslog to the Sensor and I see the traffic with tcpdump.

I see the new Remote Host now, But I don't see the syslog entries from the sensor in ELSA WEB Interface.

I ran " sudo service apache2 restart" No change
I ran "sudo service nsm restart" No change

I am querying "class=FIREWALL_ACCESS_DENY groupby:host" which shows the pfSense Remote syslog now which is great but there is no new data showing? It doesn't show any errors.

If I query "class=FIREWALL_ACCESS_DENY host=xx.xx.xx.xx" (Remote pfSense), I get the following message "Errors:No nodes available"

If I query a blocked syslog alert "61.174.51.194" in ELSA Server, i get the following message "Errors: No nodes available"

------------------------------------------------

> Is starman running on the sensor?
> pgrep -lf starman

pgrep -lf starman
2105 starman master -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
2108 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
2109 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
2110 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
2111 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
2112 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi

> Is autossh running on the sensor?
> pgrep -lf autossh

pgrep -lf autossh
3133 /usr/lib/autossh/autossh -M 0 -q -N -o ServerAliveInterval 60 -o ServerAliveCountMax 3 -i /root/.ssh/securityonion -L 3306:127.0.0.1:3306 -R 50000:localhost:3154 user...@XX.XX.XX.XX (**SENSOR**)

>
> Have you tried restarting autossh on the sensor?
> sudo pkill -USR1 autossh

sudo pkill -USR1 autossh
pgrep -lf autossh
3133 /usr/lib/autossh/autossh -M 0 -q -N -o ServerAliveInterval 60 -o ServerAliveCountMax 3 -i /root/.ssh/securityonion -L 3306:127.0.0.1:3306 -R 50000:localhost:3154 user...@xx.xx.xx.xx (**SENSOR**)

>
> Do you see the ssh tunnel on the server?
>
> pgrep -lf sshd

pgrep -lf sshd
1331 /usr/sbin/sshd -D
5605 sshd: username [priv]
5791 sshd: username
19377 sshd: username [priv]
19560 sshd: username@pts/1

[Doug Burks]

On Sat, Feb 1, 2014 at 12:07 PM, BBCan177 <bbca...@gmail.com> wrote:
> Hi Doug,
>
> I ran the commands as suggested. Autossh was running, but killing the autossh restored the second NODE in ELSA Web Interface.
>
>
> I pointed the pfSense syslog to the Sensor and I see the traffic with tcpdump.
>
> I see the new Remote Host now, But I don't see the syslog entries from the sensor in ELSA WEB Interface.
>
> I ran " sudo service apache2 restart" No change
> I ran "sudo service nsm restart" No change
>
> I am querying "class=FIREWALL_ACCESS_DENY groupby:host" which shows the pfSense Remote syslog now which is great but there is no new data showing? It doesn't show any errors.
>
> If I query "class=FIREWALL_ACCESS_DENY host=xx.xx.xx.xx" (Remote pfSense), I get the following message "Errors:No nodes available"

Have you added the pfSense parser to the sensor? If not, then the
pfSense logs won't be classified as FIREWALL_ACCESS_DENY.

[BBCan177]

> Have you added the pfSense parser to the sensor? If not, then the
> pfSense logs won't be classified as FIREWALL_ACCESS_DENY.

Yes I had previously added the parser to the Sensors patterndb file.

When I said previously that I saw the remote host in ELSA. If I change the time to after I made the changes, there is no activity.

I have also tried to query for individual IP's that are in pfSense syslogs and tried also with class=none without any success.

[Doug Burks]

Are mysql and sphinx running on the sensor?

pgrep -lf mysql

pgrep -lf sphinx

Have you tried restarting the sensor?

[BBCan177]

> pgrep -lf mysql

> pgrep -lf sphinx

pgrep -lf mysql
1581 /usr/sbin/mysqld

pgrep -lf sphinx

(No response)

> Have you tried restarting the sensor?

yes I have rebooted both server and Sensor.

[Doug Burks]

If sphinx isn't running on the sensor, that's a problem.

What happens if you try to restart sphinx?

sudo service sphinxsearch restart

Does it then show up as running?

pgrep -lf sphinx

Are you then able to connect to it?

nc localhost 9306

[BBCan177]

>
> What happens if you try to restart sphinx?

sudo service sphinxsearch restart
stop: Unknown instance:
sphinxsearch start/running, process 22824

[BBCan177]

sudo service sphinxsearch start
start: Job is already running: sphinxsearch

pgrep -lf sphinx
24048 su -s /bin/sh -c exec "$0" "$@" sphinxsearch -- /usr/bin/searchd --nodetach

nc localhost 9306

Opens and closes immediately

[Doug Burks]

Kill any running sphinx processes and then try to start sphinx manually:

su -s /bin/sh -c 'exec "$0" "$@"' sphinxsearch -- /usr/bin/searchd --nodetach

Copy any output into your reply.

[BBCan177]

> su -s /bin/sh -c 'exec "$0" "$@"' sphinxsearch -- /usr/bin/searchd --nodetach
>
> Copy any output into your reply.

su -s /bin/sh -c 'exec "$0" "$@"' sphinxsearch -- /usr/bin/searchd --nodetach

Sphinx 2.0.7-id64-dev (rel20-r3736)
Copyright (c) 2001-2012, Andrew Aksyonoff
Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file '/etc/sphinxsearch/sphinx.conf'...
WARNING: compat_sphinxql_magics=1 is deprecated; please update your application and config
listening on all interfaces, port=9306
listening on all interfaces, port=9312
precaching index 'permanent'
precaching index 'temporary'
precaching index 'real_1'
precaching index 'real_2'
precaching index 'perm_1'
precaching index 'temp_1'
precaching index 'perm_2'
precaching index 'temp_2'
precaching index 'perm_3'
precaching index 'temp_3'
precaching index 'perm_4'
precaching index 'temp_4'
precaching index 'perm_5'
precaching index 'temp_5'
precaching index 'perm_6'
precaching index 'temp_6'
precaching index 'perm_7'
precaching index 'temp_7'
precaching index 'perm_8'
precaching index 'temp_8'
precaching index 'perm_9'
precaching index 'temp_9'
precaching index 'perm_10'
precaching index 'temp_10'
precaching index 'perm_11'
precaching index 'temp_11'
precaching index 'perm_12'
precaching index 'temp_12'
precaching index 'perm_13'
precaching index 'temp_13'
precaching index 'perm_14'
precaching index 'temp_14'
precaching index 'perm_15'
precaching index 'temp_15'
precaching index 'perm_16'
precaching index 'temp_16'
precaching index 'perm_17'
precaching index 'temp_17'
precaching index 'perm_18'
precaching index 'temp_18'
precaching index 'perm_19'
precaching index 'temp_19'
precaching index 'perm_20'
precaching index 'temp_20'
precaching index 'perm_21'
precaching index 'temp_21'
precaching index 'perm_22'
precaching index 'temp_22'
precaching index 'perm_23'
precaching index 'temp_23'
precaching index 'perm_24'
precaching index 'temp_24'
precaching index 'perm_25'
precaching index 'temp_25'
precaching index 'perm_26'
precaching index 'temp_26'
precaching index 'perm_27'
precaching index 'temp_27'
precaching index 'perm_28'
precaching index 'temp_28'
precaching index 'perm_29'
precaching index 'temp_29'
precaching index 'perm_30'
precaching index 'temp_30'
precaching index 'perm_31'
precaching index 'temp_31'
precaching index 'perm_32'
precaching index 'temp_32'
precaching index 'perm_33'
precaching index 'temp_33'
precaching index 'perm_34'
precaching index 'temp_34'
precaching index 'perm_35'
precaching index 'temp_35'
precaching index 'perm_36'
precaching index 'temp_36'
precaching index 'perm_37'
precaching index 'temp_37'
precaching index 'perm_38'
precaching index 'temp_38'
precaching index 'perm_39'
precaching index 'temp_39'
precaching index 'perm_40'
precaching index 'temp_40'
precaching index 'perm_41'
precaching index 'temp_41'
precaching index 'perm_42'
precaching index 'temp_42'
precaching index 'perm_43'
precaching index 'temp_43'
precaching index 'perm_44'
precaching index 'temp_44'
precaching index 'perm_45'
precaching index 'temp_45'
precaching index 'perm_46'
precaching index 'temp_46'
precaching index 'perm_47'
precaching index 'temp_47'
precaching index 'perm_48'
precaching index 'temp_48'
precaching index 'perm_49'
precaching index 'temp_49'
precaching index 'perm_50'
precaching index 'temp_50'
precaching index 'perm_51'
precaching index 'temp_51'
precaching index 'perm_52'
precaching index 'temp_52'
precaching index 'perm_53'
precaching index 'temp_53'
precaching index 'perm_54'
precaching index 'temp_54'
precaching index 'perm_55'
precaching index 'temp_55'
precaching index 'perm_56'
precaching index 'temp_56'
precaching index 'perm_57'
precaching index 'temp_57'
precaching index 'perm_58'
precaching index 'temp_58'
precaching index 'perm_59'
precaching index 'temp_59'
precaching index 'perm_60'
precaching index 'temp_60'
precaching index 'perm_61'
precaching index 'temp_61'
precaching index 'perm_62'
precaching index 'temp_62'
precaching index 'perm_63'
precaching index 'temp_63'
precaching index 'perm_64'
precaching index 'temp_64'
precaching index 'perm_65'
precaching index 'temp_65'
precaching index 'perm_66'
precaching index 'temp_66'
precaching index 'perm_67'
precaching index 'temp_67'
precaching index 'perm_68'
precaching index 'temp_68'
precaching index 'perm_69'
precaching index 'temp_69'
precaching index 'perm_70'
precaching index 'temp_70'
precaching index 'perm_71'
precaching index 'temp_71'
precaching index 'perm_72'
precaching index 'temp_72'
precaching index 'perm_73'
precaching index 'temp_73'
precaching index 'perm_74'
precaching index 'temp_74'
precaching index 'perm_75'
precaching index 'temp_75'
precaching index 'perm_76'
precaching index 'temp_76'
precaching index 'perm_77'
precaching index 'temp_77'
precaching index 'perm_78'
precaching index 'temp_78'
precaching index 'perm_79'
precaching index 'temp_79'
precaching index 'perm_80'
precaching index 'temp_80'
precaching index 'perm_81'
precaching index 'temp_81'
precaching index 'perm_82'
precaching index 'temp_82'
precaching index 'perm_83'
precaching index 'temp_83'
precaching index 'perm_84'
precaching index 'temp_84'
precaching index 'perm_85'
precaching index 'temp_85'
precaching index 'perm_86'
precaching index 'temp_86'
precaching index 'perm_87'
precaching index 'temp_87'
precaching index 'perm_88'
precaching index 'temp_88'
precaching index 'perm_89'
precaching index 'temp_89'
precaching index 'perm_90'
precaching index 'temp_90'
precaching index 'perm_91'
precaching index 'temp_91'
precaching index 'perm_92'
precaching index 'temp_92'
precaching index 'perm_93'
precaching index 'temp_93'
precaching index 'perm_94'
precaching index 'temp_94'
precaching index 'perm_95'
precaching index 'temp_95'
precaching index 'perm_96'
precaching index 'temp_96'
precaching index 'perm_97'
precaching index 'temp_97'
precaching index 'perm_98'
precaching index 'temp_98'
precaching index 'perm_99'
precaching index 'temp_99'
precaching index 'perm_100'
precaching index 'temp_100'
precaching index 'perm_101'
precaching index 'temp_101'
precaching index 'perm_102'
precaching index 'temp_102'
precaching index 'perm_103'
precaching index 'temp_103'
precaching index 'perm_104'
precaching index 'temp_104'
precaching index 'perm_105'
precaching index 'temp_105'
precaching index 'perm_106'
precaching index 'temp_106'
precaching index 'perm_107'
precaching index 'temp_107'
precaching index 'perm_108'
precaching index 'temp_108'
precaching index 'perm_109'
precaching index 'temp_109'
precaching index 'perm_110'
precaching index 'temp_110'
precaching index 'perm_111'
precaching index 'temp_111'
precaching index 'perm_112'
precaching index 'temp_112'
precaching index 'perm_113'
precaching index 'temp_113'
precaching index 'perm_114'
precaching index 'temp_114'
precaching index 'perm_115'
precaching index 'temp_115'
precaching index 'perm_116'
precaching index 'temp_116'
precaching index 'perm_117'
precaching index 'temp_117'
precaching index 'perm_118'
precaching index 'temp_118'
precaching index 'perm_119'
precaching index 'temp_119'
precaching index 'perm_120'
precaching index 'temp_120'
precaching index 'perm_121'
precaching index 'temp_121'
precaching index 'perm_122'
precaching index 'temp_122'
precaching index 'perm_123'
precaching index 'temp_123'
precaching index 'perm_124'
precaching index 'temp_124'
precaching index 'perm_125'
precaching index 'temp_125'
precaching index 'perm_126'
precaching index 'temp_126'
precaching index 'perm_127'
precaching index 'temp_127'
precaching index 'perm_128'
precaching index 'temp_128'
precaching index 'perm_129'
precaching index 'temp_129'
precaching index 'perm_130'
precaching index 'temp_130'
precaching index 'perm_131'
precaching index 'temp_131'
precaching index 'perm_132'
precaching index 'temp_132'
precaching index 'perm_133'
precaching index 'temp_133'
precaching index 'perm_134'
precaching index 'temp_134'
precaching index 'perm_135'
precaching index 'temp_135'
precaching index 'perm_136'
precaching index 'temp_136'
precaching index 'perm_137'
precaching index 'temp_137'
precaching index 'perm_138'
precaching index 'temp_138'
precaching index 'perm_139'
precaching index 'temp_139'
precaching index 'perm_140'
precaching index 'temp_140'
precaching index 'perm_141'
precaching index 'temp_141'
precaching index 'perm_142'
precaching index 'temp_142'
precaching index 'perm_143'
precaching index 'temp_143'
precaching index 'perm_144'
precaching index 'temp_144'
precaching index 'perm_145'
precaching index 'temp_145'
precaching index 'perm_146'
precaching index 'temp_146'
precaching index 'perm_147'
precaching index 'temp_147'
precaching index 'perm_148'
precaching index 'temp_148'
precaching index 'perm_149'
precaching index 'temp_149'
precaching index 'perm_150'
precaching index 'temp_150'
precaching index 'perm_151'
precaching index 'temp_151'
precaching index 'perm_152'
precaching index 'temp_152'
precaching index 'perm_153'
precaching index 'temp_153'
precaching index 'perm_154'
precaching index 'temp_154'
precaching index 'perm_155'
precaching index 'temp_155'
precaching index 'perm_156'
precaching index 'temp_156'
precaching index 'perm_157'
precaching index 'temp_157'
precaching index 'perm_158'
precaching index 'temp_158'
precaching index 'perm_159'
precaching index 'temp_159'
precaching index 'perm_160'
precaching index 'temp_160'
precaching index 'perm_161'
precaching index 'temp_161'
precaching index 'perm_162'
precaching index 'temp_162'
precaching index 'perm_163'
precaching index 'temp_163'
precaching index 'perm_164'
precaching index 'temp_164'
precaching index 'perm_165'
precaching index 'temp_165'
precaching index 'perm_166'
precaching index 'temp_166'
precaching index 'perm_167'
precaching index 'temp_167'
precaching index 'perm_168'
precaching index 'temp_168'
precaching index 'perm_169'
precaching index 'temp_169'
precaching index 'perm_170'
precaching index 'temp_170'
precaching index 'perm_171'
precaching index 'temp_171'
precaching index 'perm_172'
precaching index 'temp_172'
precaching index 'perm_173'
precaching index 'temp_173'
precaching index 'perm_174'
precaching index 'temp_174'
precaching index 'perm_175'
precaching index 'temp_175'
precaching index 'perm_176'
precaching index 'temp_176'
precaching index 'perm_177'
precaching index 'temp_177'
precaching index 'perm_178'
precaching index 'temp_178'
precaching index 'perm_179'
precaching index 'temp_179'
precaching index 'perm_180'
precaching index 'temp_180'
precaching index 'perm_181'
precaching index 'temp_181'
precaching index 'perm_182'
precaching index 'temp_182'
precaching index 'perm_183'
precaching index 'temp_183'
precaching index 'perm_184'
precaching index 'temp_184'
precaching index 'perm_185'
precaching index 'temp_185'
precaching index 'perm_186'
precaching index 'temp_186'
precaching index 'perm_187'
precaching index 'temp_187'
precaching index 'perm_188'
precaching index 'temp_188'
precaching index 'perm_189'
precaching index 'temp_189'
precaching index 'perm_190'
precaching index 'temp_190'
precaching index 'perm_191'
precaching index 'temp_191'
precaching index 'perm_192'
precaching index 'temp_192'
precaching index 'perm_193'
precaching index 'temp_193'
precaching index 'perm_194'
precaching index 'temp_194'
precaching index 'perm_195'
precaching index 'temp_195'
precaching index 'perm_196'
precaching index 'temp_196'
precaching index 'perm_197'
precaching index 'temp_197'
precaching index 'perm_198'
precaching index 'temp_198'
precaching index 'perm_199'
precaching index 'temp_199'
precaching index 'perm_200'
precaching index 'temp_200'
precached 404 indexes in 1.931 sec
binlog: replaying log /var/lib/sphinxsearch/data/binlog.001
FATAL: binlog: log open error: failed to open /var/lib/sphinxsearch/data/binlog.001: No such file or directory

-------------------------

pgrep -lf sphinx

(nothing)

sudo service sphinxsearch start
sphinxsearch start/running, process 4217

nc localhost 9306

(nothing)

-------------------------

sudo service apache2 restart
* Restarting web server apache2 apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1 for ServerName
... waiting ...apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1 for ServerName
[ OK ]

nc localhost 9306

(nothing)

sudo service sphinxsearch status
sphinxsearch start/running, process 1669

[Doug Burks]

On Sat, Feb 1, 2014 at 4:03 PM, BBCan177 <bbca...@gmail.com> wrote:
> binlog: replaying log /var/lib/sphinxsearch/data/binlog.001
> FATAL: binlog: log open error: failed to open /var/lib/sphinxsearch/data/binlog.001: No such file or directory

What's the output of the following?

ls -alh /var/lib/sphinxsearch/data/


--
Doug Burks

[BBCan177]

> What's the output of the following?

> ls -alh /var/lib/sphinxsearch/data/

ls -alh /var/lib/sphinxsearch/data/
total 12K
drwxr-xr-x 2 sphinxsearch root 4.0K Feb 1 21:06 .
drwxr-xr-x 3 sphinxsearch root 4.0K Dec 18 2012 ..
-rw------- 1 sphinxsearch sphinxsearch 0 Feb 1 21:06 binlog.lock
-rw------- 1 sphinxsearch sphinxsearch 11 Jan 25 22:39 binlog.meta

[Doug Burks]

Permissions look correct. Any idea why you'd be missing binlog.001?

Perhaps try the following:

sudo mv /var/lib/sphinxsearch/data/binlog.* /tmp

sudo su -s /bin/sh -c 'exec "$0" "$@"' sphinxsearch --
/usr/bin/searchd --nodetach

[BBCan177]

> Perhaps try the following:

> sudo mv /var/lib/sphinxsearch/data/binlog.* /tmp

> sudo su -s /bin/sh -c 'exec "$0" "$@"' sphinxsearch -- /usr/bin/searchd --nodetach

sudo mv /var/lib/sphinxsearch/data/binlog.* /tmp

sudo su -s /bin/sh -c 'exec "$0" "$@"' sphinxsearch -- /usr/bin/searchd --nodetach

Sphinx 2.0.7-id64-dev (rel20-r3736)
Copyright (c) 2001-2012, Andrew Aksyonoff
Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file '/etc/sphinxsearch/sphinx.conf'...
WARNING: compat_sphinxql_magics=1 is deprecated; please update your application and config

FATAL: failed to lock pid file '/var/run/sphinxsearch/searchd.pid': Resource temporarily unavailable (searchd already running?)

[BBCan177]

nc localhost 9306, window now opens properly.

But still no results in ELSA WEB Server.

Also tried to "sudo service apache2 restart"

[Doug Burks]

So did you receive the "FATAL: failed to lock pid file
'/var/run/sphinxsearch/searchd.pid'" error? If so, please try the
following:

- reboot the sensor

- wait a few minutes

- verify that starman, mysql, sphinx, and autossh are running:
pgrep -lf starman

pgrep -lf mysql
pgrep -lf sphinx

pgrep -lf autossh

- verify that you can connect to mysql and sphinx:
nc localhost 50000
nc localhost 9306

- go to your ELSA web page and shift-reload, verify that it shows 2 nodes

- try your search again

[Doug Burks]

"receive" should have been "resolve"

So did you resolve the "FATAL: failed to lock pid file

'/var/run/sphinxsearch/searchd.pid'" error? If so, please try the
following:

- reboot the sensor

- wait a few minutes

- verify that starman, mysql, sphinx, and autossh are running:
pgrep -lf starman
pgrep -lf mysql
pgrep -lf sphinx
pgrep -lf autossh

- verify that you can connect to mysql and sphinx:
nc localhost 50000
nc localhost 9306

- go to your ELSA web page and shift-reload, verify that it shows 2 nodes

- try your search again

--
Doug Burks

[BBCan177]

Hi Doug,

Issue has been resolved. Sometimes you need to walk away and grab a few beers!

Could the status of Starman, mysql, Sphinx and autossh be added to SOSTAT? or something similar?

I have attached a txt document outlining the commands used to fix my issue.

Hope it might help someone else.

Thanks for your patience. Some tears were shed on this onion!

[Doug Burks]

Was there anything else you did to resolve other than reboot?

I'll consider adding more verbose output to sostat as time allows. 

--
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at http://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/groups/opt_out.

--
Doug Burks

[BBCan177]

On Sunday, February 2, 2014 8:27:27 AM UTC-5, Doug Burks wrote:
> Was there anything else you did to resolve other than reboot?

The first issue was autossh showed a connection, so when I ran
"sudo pkill -USR1 autossh" the remote sensor node showed up in ELSA web Interface.

Sphinx wasn't running. Not sure why it was down, But starting it with
"sudo service sphinxsearch start"

Ran
sudo mv /var/lib/sphinxsearch/data/binlog.* /tmp

su -s /bin/sh -c 'exec "$0" "$@"' sphinxsearch -- /usr/bin/searchd --nodetach

At this point "nc localhost 9306" worked, but no records found in ELSA.

I came back hours later and found the "binlog.001" in the folder.

I ran all the pgrep's and nc commands (all ok)

Rebooted both server/sensor

ELSA began to show the Sensors ELSA logs.

> I'll consider adding more verbose output to sostat as time allows. 

Thanks.

[BBCan177]

Trying to query ELSA WEB interface this morning.

2 Nodes show up, but when the following queries return no results -

Connections (all queries empty)
DNS all empty(except for Top nxdomain, results from 15hrs ago)
HTTP all empty
FTP all empty

pgrep -lf mysql
23550 /usr/sbin/mysqld

Looks like its missing the last few days of data? And no new data.

Tried restarting mysql, sudo service nsm stop/start.
sudo service apache2 restart

nc localhost 50000 (Doesnt load)
nc localhost 3306 (ok)
nc localhost 9306 (ok)

OSSEC and Snort logs are reporting current activity ok. I rdp'd to some servers to generate traffic and its showing up in SGUIL, but nothing in ELSA. Firewall_ACCESS_DENY returns current results without issue.

[BBCan177]

Changing the elsa_web.conf "Query timeout from 30 to 10000", and Restarting Apache, now returns data from 20hrs ago.

I tried to increase to 20000 with no difference?

[Doug Burks]

On Sun, Feb 2, 2014 at 5:55 PM, BBCan177 <bbca...@gmail.com> wrote:
> Changing the elsa_web.conf "Query timeout from 30 to 10000", and Restarting Apache, now returns data from 20hrs ago.
> I tried to increase to 20000 with no difference?

Did you make this change on both boxes?

What do you get in the yellow pop-up when hovering over the From and
To date boxes?

[BBCan177]

On Monday, February 3, 2014 6:33:31 AM UTC-5, Doug Burks wrote:

> Did you make this change on both boxes?

Why do i keep forgetting to update both boxes!
"Insanity is doing the same thing over and over expecting different results" lol

> What do you get in the yellow pop-up when hovering over the From and
> To date boxes?

From
"Earliest:2013-11-07 02:27:21, Archive Earliest:2013-10-27 18:21:44"

To
"Earliest:2014-02-03 14:20:01, Archive Earliest:2014-01-28 00:02:16"

I have changed both boxes to 10000, restart apache2 on both, reloaded ELSA Web interface but still cant query anything after this date/time

2014-02-02 04:00:01

[Doug Burks]

On Mon, Feb 3, 2014 at 9:39 AM, BBCan177 <bbca...@gmail.com> wrote:
> I have changed both boxes to 10000, restart apache2 on both,

Sensor uses starman instead of apache2.

--
Doug Burks

[BBCan177]

> Sensor uses starman instead of apache2.

Thanks Doug,

Stop/Start Starman on the sensor didn't fix the issue. Rebooted both server/sensor and it seems to be returning the correct queries now.

Its a little bit of a nuisance to use the new ELSA Web page if you want to use a custom date/time for different lookups as it keeps defaulting to the default date/time.

I will say that I have seen a big improvement in system speed overall.

[Doug Burks]

On Mon, Feb 3, 2014 at 10:16 AM, BBCan177 <bbca...@gmail.com> wrote:
>
>> Sensor uses starman instead of apache2.
>
> Thanks Doug,
>
> Stop/Start Starman on the sensor didn't fix the issue. Rebooted both server/sensor and it seems to be returning the correct queries now.
>
> Its a little bit of a nuisance to use the new ELSA Web page if you want to use a custom date/time for different lookups as it keeps defaulting to the default date/time.

Have you tried changing the default_start_time_offset value in
/etc/elsa_web.conf?

Alternatively, you could rewrite /var/www/elsa/menu.php to use
Javascript to populate the search instead of forcing a page refresh to
submit the search via URL.

> I will say that I have seen a big improvement in system speed overall.

Glad to hear it!


--
Doug Burks