ntopng on SO install guide

[Greg Williams]

Thought since I needed ntopng for some other non-technical people on my team, I thought post the installation instructions here.

In my opinion, ntopng on SO is just to provide a graphical instantaneous snapshot of current flows and meant to be used as a supplementary tool to Bro connection logs and Argus, which keep more historical data. With ntopng, you can see current flows to Netflix, YouTube, BitTorrent, etc without digging into the data. Ntopng also uses very little resources.

I have zero packet loss from ntopng after running this and .0001% packet loss on my Bro processes. With ZeroMQ, both Bro and Ntopng share the packets from pf_ring.

Note, I'm only running Bro and ntopng on this system.

apt-get install redis-server rrdtool libzmq-dev build-essential
cd /usr/local/src
wget http://www.nmon.net/packages/ubuntu/x64/ntopng/ntopng_1.1.1-7131_amd64.deb
dpkg --install ntopng_1.1.1-7131_amd64.deb
vi /etc/ntopng/ntopng.start
--local-networks "xxx.xxx.0.0/16"
--interface 1
--daemon
--dns-mode 1
vi /etc/ntopng/ntopng.conf
-G=/var/run/ntopng.pid
wget http://download.zeromq.org/zeromq-3.2.4.tar.gz
tar xzvf zeromq-3.2.4.tar.gz
cd zeromq-3.2.4.tar.gz
./configure
make && make install
vi /etc/ld.so.conf
/usr/local/lib
ldconfig
cd ..
wget http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNumv6.dat.gz
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCityv6-beta/GeoLiteCityv6.dat.gz
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
wget http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz
gunzip Geo*
mv Geo*.dat /usr/local/share/ntopng/httpdocs/geoip/
ufw allow from xxx.xxx.xxx.xxx to any port 3000 proto tcp
/etc/init.d/ntopng start

Make sure you change the username/password for ntopng from admin/admin to something else.

Hope you enjoy!

=========================================================================
pf_ring stats
=========================================================================
Appl. Name : ntopng
Tot Packets : 168217588
Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Appl. Name : <unknown>
Tot Packets : 27968256
Tot Pkt Lost : 22
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Appl. Name : <unknown>
Tot Packets : 32436165
Tot Pkt Lost : 29
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Appl. Name : <unknown>
Tot Packets : 25166047
Tot Pkt Lost : 23
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Appl. Name : <unknown>
Tot Packets : 26893841
Tot Pkt Lost : 27
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Appl. Name : <unknown>
Tot Packets : 26135909
Tot Pkt Lost : 23
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Appl. Name : <unknown>
Tot Packets : 27267537
Tot Pkt Lost : 49
TX: Send Errors : 0
Reflect: Fwd Errors: 0

[Doug Burks]

Hi Greg,

Thanks for sharing!

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/groups/opt_out.



--
Doug Burks

[Rob Pelletier]

Thanks for the post, this is great. However, does anyone know why ntopng.start would be ignored? None of my configurations from ntopng.start are honored when I run ntopng.

[Greg Williams]

On Friday, January 3, 2014 6:26:25 AM UTC-7, Rob Pelletier wrote:
> Thanks for the post, this is great. However, does anyone know why ntopng.start would be ignored? None of my configurations from ntopng.start are honored when I run ntopng.

What does your ntopng.start look like? What are the permissions? How are you starting ntopng? Just want to make sure

[Rob Pelletier]

I have used several different tutorials that all go about it basically the same way. I used your tutorial to install ntopng, although I have tried other with the same result.

I came across your tutorial and loved that it gave me a nice upstart script, which I had been lacking in other installations (I have previously been adding the startup to /etc/rc.local) When using rc.local I previously just added all the switches there, which made using ntopng.start unnecessary. However, I would much rather use ntopng.start, but have never been able to get it to work.

ntopng.start looks like this (I have masked my actual network ip range) Permissions are 640

--local-networks xxx.xxx.xxx.0/21
--interface 2
--http-port 3100
--daemon

I use /etc/init.d/ntopng start to start it, but it simply does not respect the parameters I have placed above.

Thank you for you assistance with this.

[Greg Williams]

Not sure if this will have any impact, but try setting your permissions to 644 and put quotes around your local-networks.

--local-networks "xxx.xxx.xxx.0/21"

[Rob Pelletier]

On Friday, January 3, 2014 11:45:31 AM UTC-5, Greg Williams wrote:
> Not sure if this will have any impact, but try setting your permissions to 644 and put quotes around your local-networks.
>
> --local-networks "xxx.xxx.xxx.0/21"

No change, still nothing

[Martin Paszkiewicz]

Greg, thank you for sharing your installation instructions. I also have the same problem Rob has described. ntopng runs fine when I start it manually with all the parameters but does not start with ntopng.start automatically. What worked for me is passing all the switches in ntopng.start via ntopng.conf

Seems to work fine for me.

Good piece of info for adding users: https://svn.ntop.org/svn/ntop/trunk/ntopng/README.users

Kind Regards,
Martin Paszkiewicz

[Greg Williams]

Hmmm, interesting. I'll have to try my install again and see what changed. I didn't have to modify anything other than what I put in the instructions. Thank you for sharing!

[Kevin Branch]

Greg, thanks for this guide. It really helped me get ntopng up and running on SO.

Here are a few revisions I made along the way so it would work for me:

1. /etc/ntopng/ntopng.start is just a flag file which if present will result in ntopng running at boot time. The actual content of the file is not used at all, at least according to my review of /etc/init.d/ntopng and the ntopng man page. Thus all config options need to be put into /etc/ntopng/ntopng.conf

2. Any of the options in /etc/ntopng/ntopng.conf which take a value must have an equal sign rather than a space between the option at the value, like this:
--dns-mode=1
Otherwise the option seems to be ignored.

3. ntopng runs by default as nobody, and nobody does not have write permission to /var/run so we can't put the pid file there. It defaults to using /var/tmp for the pid file location so I use this in ntopng.conf for pid path:
-G=/var/tmp/ntopng.pid

4. Unless you are using nprobe to send flows to ntopng, I don't think zeromq is needed at all. With or without zeromq installed, a separate PF_RING ring is allocated for ntop, so I skip zeromq and its dependencies.

Here is the procedure I'm using now for installing ntopng on SO:

apt-get install redis-server rrdtool
cd /usr/local/src
# Get the latest ntopng deb package from http://www.nmon.net/packages/ubuntu/x64/ntopng/. This apepars to change daily. Adjust deb filenames in following 2 lines accordingly.
wget http://www.nmon.net/packages/ubuntu/x64/ntopng/ntopng_1.1.2-7258_amd64.deb
dpkg --install ntopng_1.1.2-7258_amd64.deb
touch /etc/ntopng/ntopng.start
mkdir /usr/local/ntopng
chown nobody:root /usr/local/ntopng
# Edit /etc/ntopng/ntopng.conf, making sure it at least contains "-G=/var/tmp/ntopng.pid" which is used by the control script. Use the ntopng man page to figure out what options you want. I use something like this:
--data-dir=/usr/local/ntopng
--local-networks="192.168.0.0/16,10.0.0.0/8"
--interface=eth1
--dns-mode=1
--disable-login
--packet-filter="ip and not proto ipv6 and not ether host ff:ff:ff:ff:ff:ff and not net (224.0.0.0/8 or 239.0.0.0/8)"
--daemon
-G=/var/tmp/ntopng.pid
# Make sure you use -G instead of its long name (--pid-path) in the above file. The control script looks for a literal "-G=" line in the conf file and if you use --pid-path instead, the script will not be able to find the pid file.

wget http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNumv6.dat.gz
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCityv6-beta/GeoLiteCityv6.dat.gz
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
wget http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz
gunzip Geo*
mv Geo*.dat /usr/local/share/ntopng/httpdocs/geoip/

ufw allow 3000/tcp
service ntopng start

Thanks again for taking the time to write things up. You saved me a bunch of time.

Kevin

[Wayne Veilleux]

FYI Kevin, there is also a deb package containing all the GeoIP files at

http://www.nmon.net/packages/ubuntu/all/ntopng-data/ (daily updated)

Wayne

[Kevin Branch]

Thanks, Wayne.

I just tried that out and it looks great. I've updated the Ntopng
deployment procedure accordingly and published it in the Wiki at
https://code.google.com/p/security-onion/wiki/DeployingNtopng

Doug, would you like me to put a reference to this new Wiki article at
the bottom of the "Tips and Tricks" section?

Kevin

[Doug Burks]

On Thu, Jun 5, 2014 at 10:11 AM, Kevin Branch
<branchnet...@gmail.com> wrote:
> Doug, would you like me to put a reference to this new Wiki article at the
> bottom of the "Tips and Tricks" section?

Sounds great, thanks!

[Wayne Veilleux]

Kevin, Doug,

Why not include this great ntopng tool in SO distro ?

Wayne

[Doug Burks]

On Thu, Jun 5, 2014 at 10:23 AM, Wayne Veilleux
<wayne.v...@gmail.com> wrote:
> Kevin, Doug,
>
> Why not include this great ntopng tool in SO distro ?

Hi Wayne,

We've had Issue 96 open since 2011:
https://code.google.com/p/security-onion/issues/detail?id=96

Unfortunately, we don't have the manpower to support ntopng right now.
If somebody would like to volunteer to take that on, we'll consider
it!

Thanks,
Doug

[Wayne Veilleux]

I just contact Luca (the ntopng maintainer) about that. I'll get back to you with his thoughts.
Should it be compiled and re-packaged into SO meta-package ? (I know his to compile from source ntopng on SO) or use the deb package from Luca repo ?
--
Wayne

[Wayne Veilleux]

Doug,
I just got a positive answer from Luca Deri (the maintainer of ntopng) and he would happy to have ntopng integrated into SO and I'm willing to be in charge of it. But, that will be the first time doing this. As I told you, I know how to patch, compile, build packages and also do some shell script but I'm not aware on how to work into your community project. All I need is documentation on how to do it and I'm ready and happy to learn it ;-)
--
Wayne

[Doug Burks]

Hi Wayne,

Here's what a possible ntopng integration in Security Onion would look like:

- ntopng would be optional, not required
- 32-bit and 64-bit packages in our PPA
- compiled against PF_RING
- updates to Setup for configuration
- updates to NSM scripts for handling processes
- What does a full enterprise deployment of ntopng look like? If I
have a 10-sensor deployment, how do I view ntop data from all of them
in one web interface?

To get started, you can create your own PPA and begin building your
own packages there using Ubuntu's documentation:
https://help.launchpad.net/Packaging/PPA

Thanks,
Doug

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.

> For more options, visit https://groups.google.com/d/optout.



--
Doug Burks

[Wayne Veilleux]

Replies inline...

--

Wayne

Le 2014-06-06 à 07:39, Doug Burks <doug....@gmail.com> a écrit :

Hi Wayne,

Here's what a possible ntopng integration in Security Onion would look like:

- ntopng would be optional, not required

Yes, this is also what I though.

- 32-bit and 64-bit packages in our PPA

OK

- compiled against PF_RING

If I’m not wrong, PF_FING come from ntop project, so that’s for sure it is working with PF_RING

- updates to Setup for configuration

Yes

- updates to NSM scripts for handling processes

Yes

- What does a full enterprise deployment of ntopng look like?  If I
have a 10-sensor deployment, how do I view ntop data from all of them
in one web interface?

ntopng fit perfectly with SO architecture because you can fetch ntop informations from a central server to each ntopng instance running on a sensor on the monitoring network interface. The central manager can also run an ntopng monitoring instance on a local ethX like our "standalone » setup. ntopng also maintain each flow (byte-in,byte-out,ip-protocol,apps,vlan_id,….) of each host it see on the monitoring interface and I think we should have a look to use it also like a host asset manager. Do you know OSSIM from AlienVault also include ntop ?

To get started, you can create your own PPA and begin building your
own packages there using Ubuntu's documentation:
https://help.launchpad.net/Packaging/PPA

I will to read this and start playing with it.

You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/PycIs8P7Tbg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.

[Doug Burks]

On Mon, Jun 9, 2014 at 3:54 PM, Wayne Veilleux <wayne.v...@gmail.com> wrote:
> - compiled against PF_RING
>
>
> If I’m not wrong, PF_FING come from ntop project, so that’s for sure it is
> working with PF_RING

I believe ntop can be compiled against the standard libpcap library or
against PF_RING, our ntop packages would need to be compiled using our
PF_RING packages.


--
Doug Burks

[Wayne Veilleux]

Doug,
I already compiled ntopng directly on a SO 12.04.4 distro with these extra package installed to compile it:

1. sudo apt-get install build-essential checkinstall
2. sudo apt-get install cvs subversion git-core mercurial dh-autoreconf
3. sudo apt-get install redis-server ntp libglib2.0-dev libgeoip-dev libgeoip1 sqlite3 libsqlite3-0 libsqlite3-dev libpcap-dev librrd-dev rrdtool
4. Download and compile ntopng:
svn co https://svn.ntop.org/svn/ntop/trunk/ntopng/
cd ntopng
./autogen.sh
./configure
make
make geoip
sudo make install

—
Wayne

[Doug Burks]

If you just did a ./configure by itself, my guess is that it compiled
against standard libpcap. Take a look at some of our PF_RING-aware
applications like snort, suricata, and Bro to see how they are
compiled against PF_RING.

On Tue, Jun 10, 2014 at 6:53 AM, Wayne Veilleux
<wayne.v...@gmail.com> wrote:
> Doug,
> I already compiled ntopng directly on a SO 12.04.4 distro with these extra package installed to compile it:
>
> 1. sudo apt-get install build-essential checkinstall
> 2. sudo apt-get install cvs subversion git-core mercurial dh-autoreconf
> 3. sudo apt-get install redis-server ntp libglib2.0-dev libgeoip-dev libgeoip1 sqlite3 libsqlite3-0 libsqlite3-dev libpcap-dev librrd-dev rrdtool
> 4. Download and compile ntopng:
> svn co https://svn.ntop.org/svn/ntop/trunk/ntopng/
> cd ntopng
> ./autogen.sh
> ./configure
> make
> make geoip
> sudo make install
>

> --

> Wayne
>
>
>
> Le 2014-06-10 à 06:50, Doug Burks <doug....@gmail.com> a écrit :
>
>> On Mon, Jun 9, 2014 at 3:54 PM, Wayne Veilleux <wayne.v...@gmail.com> wrote:
>>> - compiled against PF_RING
>>>
>>>
>>> If I'm not wrong, PF_FING come from ntop project, so that's for sure it is
>>> working with PF_RING
>>
>> I believe ntop can be compiled against the standard libpcap library or
>> against PF_RING, our ntop packages would need to be compiled using our
>> PF_RING packages.
>>
>>
>> --
>> Doug Burks
>>
>> --
>> You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/PycIs8P7Tbg/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.
>> To post to this group, send email to securit...@googlegroups.com.
>> Visit this group at http://groups.google.com/group/security-onion.
>> For more options, visit https://groups.google.com/d/optout.
>
> --

> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.

> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.

--
Doug Burks

[Wayne Veilleux]

I think you’re right:

ldd /usr/local/bin/ntopng|grep pcap
libpcap.so.0.8 => /usr/lib/x86_64-linux-gnu/libpcap.so.0.8 (0x00007f4f9101f000)

and I see:
ldd /usr/bin/snort| grep ring
libpcap.so.1 => /opt/pfring/lib/libpcap.so.1 (0x00007fe60bc35000)

I’ll have a look on this and I’ll get back to you.

Thanks Doug.
—
Wayne

[Wayne Veilleux]

Doug,
I see the so meta-package securityonion-pfring-userland is installed and the files are in the /opt/pfring directory. Is there a dev package I need to install to compile ntopng with it ? Or all the dev an headers libraries are already include ?
—
Wayne

[Wayne Veilleux]

Forget this Doug (sorry for that), I found it securityonion-pfring-userland-devel :)
—
Wayne

[Doug Burks]

Take a look at one of our PF_RING-aware applications like snort,
suricata, or bro:

apt-get source securityonion-snort

In the debian directory, you should a couple of files of interest like
control and rules. They tell Launchpad the build dependencies and how
to compile the package.

On Tue, Jun 10, 2014 at 8:54 AM, Wayne Veilleux
<wayne.v...@gmail.com> wrote:
> Doug,

> I see the so meta-package securityonion-pfring-userland is installed and the files are in the /opt/pfring directory. Is there a dev package I need to install to compile ntopng with it ? Or all the dev an headers libraries are already include ?

[Wayne Veilleux]

Doug,

I have now compiled and I can run ntopng with the pf_ring library from SO. Next step, PPA. But before, I would like to talk to Luca Deri to ask him to include the patch I made on the configure.ac file to use the pf_ring library from SO.
—
Wayne

[Doug Burks]

OK, thanks for the update!

On Wed, Jun 11, 2014 at 2:57 PM, Wayne Veilleux
<wayne.v...@gmail.com> wrote:
> Doug,
>
> I have now compiled and I can run ntopng with the pf_ring library from SO. Next step, PPA. But before, I would like to talk to Luca Deri to ask him to include the patch I made on the configure.ac file to use the pf_ring library from SO.

[Wayne Veilleux]

Doug,
We've compiled ntopng with pf_ring libs in /opt/pfring in SO Distro and this patch will be maintain in the source (so new version will also works), but we're having a serious problem trying to upload the source code to Launchpad from the subversion (svn) version control system. I'm in contact with the ntopng developers and they are really willing to integrate ntopng within the SO project. We just upgrade the svn server hoping that would work with the Launchpad import svn tool (which using bazaar svn import in the background) and it is always failed. So, my question: is there someone on your SO team that can help on that issue ? I google the failed error message Launchpad give and I thing we're not alone having problem with this issue.
Regards,
Wayne

[Doug Burks]

Hi Wayne,

I don't have any experience uploading to Launchpad from svn. All of
my packages start with a tarball which I then build into a source
package and upload to Launchpad using dput.

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.



--
Doug Burks

http://securityonionsolutions.com

[Peter Manev]

On Wednesday, July 23, 2014 4:46:42 PM UTC+2, Wayne Veilleux wrote:
> Doug,
>
> We've compiled ntopng with pf_ring libs in /opt/pfring in SO Distro and this patch will be maintain in the source (so new version will also works), but we're having a serious problem trying to upload the source code to Launchpad from the subversion (svn) version control system. I'm in contact with the ntopng developers and they are really willing to integrate ntopng within the SO project. We just upgrade the svn server hoping that would work with the Launchpad import svn tool (which using bazaar svn import in the background) and it is always failed. So, my question: is there someone on your SO team that can help on that issue ? I google the failed error message Launchpad give and I thing we're not alone having problem with this issue.

Hi Wayne,

Can you share in a bit more detail what is the problem?

You can take that offline (off the list since I suspect it could be a long thread) if you would like and if/when we figure it out we could post the solution back on the list.

The import is the first easy step from my experience, the hardest part is after that :)